Integrations on SILO

Using MinIO with Veeam

Mon, 01 Jan 0001 00:00:00 +0000

When using Veeam Backup and Replication, you can use S3 compatible object storage such as MinIO as a capacity tier for backups. This disaggregates storage for the Veeam infrastructure and allows you to retain control of your data. With the ease of use of setup and administration of MinIO, it allows a Veeam backup admin to easily deploy their own object store for capacity tiering.

Prerequisites

One or both of Veeam Backup and Replication with support for S3 compatible object store (e.g. 9.5.4) and Veeam Backup for Office365 (VBO)
MinIO object storage set up per https://silo.pigsty.io/index.html#procedure
Veeam requires TLS connections to the object storage. This can be configured per https://silo.pigsty.io/operations/network-encryption.html
The S3 bucket, Access Key and Secret Key have to be created before and outside of Veeam.
Configure the MinIO Client for the Veeam MinIO endpoint - https://silo.pigsty.io/reference/minio-mc.html

Setting up an S3 compatible object store for Veeam Backup and Replication

Create a bucket for Veeam backups

Create a bucket for Veeam Backup, e.g.,

Disaggregated HDP Spark and Hive with MinIO

Mon, 01 Jan 0001 00:00:00 +0000

1. Cloud-native Architecture

Kubernetes manages stateless Spark and Hive containers elastically on the compute nodes. Spark has native scheduler integration with Kubernetes. Hive, for legacy reasons, uses YARN scheduler on top of Kubernetes.

All access to MinIO object storage is via S3/SQL SELECT API. In addition to the compute nodes, MinIO containers are also managed by Kubernetes as stateful containers with local storage (JBOD/JBOF) mapped as persistent local volumes. This architecture enables multi-tenant MinIO, allowing isolation of data between customers.

AWS CLI with MinIO Server

Mon, 01 Jan 0001 00:00:00 +0000

AWS CLI is a unified tool to manage AWS services. It is frequently the tool used to transfer data in and out of AWS S3. It works with any S3 compatible cloud storage service.

In this recipe we will learn how to configure and use AWS CLI to manage data with MinIO Server.

1. Prerequisites

Install MinIO Server from here.

2. Installation

Install AWS CLI from https://aws.amazon.com/cli/

3. Configuration

To configure AWS CLI, type aws configure and specify the MinIO key information.

Configure NGINX Proxy for MinIO Server

Mon, 01 Jan 0001 00:00:00 +0000

The following documentation provides a baseline for configuring NGINX to proxy requests to MinIO in a Linux environment. It is not intended as a comprehensive approach to NGINX, proxying, or reverse proxying in general. Modify the configuration as necessary for your infrastructure.

This documentation assumes the following:

An existing NGINX deployment
An existing MinIO deployment
A DNS hostname which uniquely identifies the MinIO deployment

There are two models for proxying requests to the MinIO Server API and the MinIO Console:

Upload Files Using Pre-signed URLs

Mon, 01 Jan 0001 00:00:00 +0000

Using pre-signed URLs, a client can upload files directly to an S3-compatible cloud storage server (S3) without exposing the S3 credentials to the user.

This guide describes how to use the presignedPutObject API from the MinIO JavaScript Library to generate a pre-signed URL. This is demonstrated through a JavaScript example in which an Express Node.js server exposes an endpoint to generate a pre-signed URL and a client-side web application uploads a file to MinIO Server using that URL.

Generate Let’s Encrypt certificate using Certbot for MinIO

Mon, 01 Jan 0001 00:00:00 +0000

Let’s Encrypt is a new free, automated, and open source, Certificate Authority.

Certbot is a console based certificate generation tool for Let’s Encrypt.

In this recipe, we will generate a Let’s Encypt certificate using Certbot. This certificate will then be deployed for use in the MinIO server.

1. Prerequisites

Install MinIO Server from here.
Install Certbot from here

2. Dependencies

Port 443 for https needs to be open and available at time of executing certbot.
Certbot needs root access while executing because only root is allowed to bind to any port below 1024.
We will be using our own domain myminio.com as an example in this recipe. Replace with your own domain under your setup.

3. Recipe Steps

Step 1: Install Certbot

Install Certbot by following the documentation at https://certbot.eff.org/