ScyllaDB is a high-performance NoSQL database that is designed for real-time big data applications and workloads.

Install scylladb on 1 ec2 -> provides service for better throughput and low latency

setup configuration files for a better communication within cluster

run the scylladb service

cql shell -> to interact with the database

Create a keyspace and define tables using CQL. Data modeling is essential to structure your data effectively.

have a java or a python application and connect with scylladb using scylla-driver

Write or modify your application code to interact with ScyllaDB. You can use client libraries like scylla-driver for Python, Java, or other languages.

scaling nodes -> add nodes in cluster for high availibility and provide details in configuration files accordingly

use promethius , grafana for monitoring

implement backup for disaster recovery

---

scylla db on ubuntu 22, t2 medium

22Gb storage

scylladb need java8

sudo apt update
sudo apt-get install -y openjdk-8-jre-headless
sudo update-java-alternatives --jre-headless -s java-1.8.0-openjdk-amd64
sudo mkdir -p /etc/apt/keyrings
sudo gpg --homedir /tmp --no-default-keyring --keyring /etc/apt/keyrings/scylladb.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys d0a112e067426ab2
sudo wget -O /etc/apt/sources.list.d/scylla.list http://downloads.scylladb.com/deb/debian/scylla-5.2.list
sudo apt update
sudo apt-get install -y scylla
sudo scylla_setup
OR
sudo scylla_dev_mode_setup --developer-mode 1

Select all yes , no press enter (enter meaning is no)

in the last step press enter where it is stuck

sudo systemctl start scylla-server

nodetool status

5432,9042,6379

cqlsh -u cassandra -p cassandra

SELECT * FROM system_auth.roles; -> has no password associated with cassandra which is a superuser

try to change password for cassandra

ALTER ROLE cassandra WITH PASSWORD = 'password';

Unauthorized: Error from server: code=2100 [Unauthorized] message="You have to be logged in and not anonymous to perform this request"

sudo systemctl stop scylla-server

add in /etc/scylla/scylla.yaml

sudo vim /etc/scylla/scylla.yaml

1. authenticator: org.apache.cassandra.auth.PasswordAuthenticator

2. authorizer: org.apache.cassandra.auth.CassandraAuthorizer

sudo systemctl start scylla-server

cqlsh -u cassandra -p cassandra

cqlsh -u cassandra -p cassandra 10.1.3.4 9042

ALTER ROLE cassandra WITH PASSWORD = 'password';

CREATE KEYSPACE employee_db WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

CREATE ROLE scylladb WITH PASSWORD = 'password' AND LOGIN = true;

GRANT CREATE ON KEYSPACE employee_db TO scylladb;

exit

cqlsh -u cassandra -p password 10.1.3.4 9042

sudo apt install git -y

sudo apt update

vim config.yaml

127.0.0.1 -> host

vim migration.json

{

"database": "cassandra://127.0.0.1:9042/employee_db?user.."

}

---

cd ..

wget https://github.com/golang-migrate/migrate/releases/download/v4.16.2/migrate.linux-amd64.tar.gz

tar -xvzf migrate.linux-amd64.tar.gz

sudo mv migrate /usr/bin/migrate

sudo apt install make

sudo apt-get install jq -y

---

employeedb

sudo apt update

sudo apt install golang -y (rather install go version 1.21 from google)

git clone https://github.com/OT-MICROSERVICES/employee-api.git

cd employee-api

migrate -source file://migration -database "$(cat migration.json | jq -r '.database')" up

go mod tidy

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o employee-api main.go

./employee-api

http://18.226.169.1:8080/api/v1/employee/search/all

{
  "id": "12347",
  "name": "Shubham",
  "designation": "Software Engineer",
  "department": "Engineering",
  "joining_date": "2023-10-15",
  "address": "123 Main Street, City, Country",
  "office_location": "Office A",
  "status": "Active",
  "email": "john.doe@example.com",
  "phone_number": "+1 (123) 456-7890"
}

---

salary

sudo apt update

sudo apt install maven -y

sudo apt install openjdk-17-jdk -y

git clone https://github.com/OT-MICROSERVICES/salary-api.git

cd salary-api

sudo update-alternatives --config java

scylla requires java 8, salaryapi requires java 17

migration.json, src/main/resources/application.yml, src/test/resources/application.yml

migrate -source [file://migration](file://migration) -database "$(cat migration.json | jq -r '.database')" up

mvn clean install package -DskipTests

java -jar target/*.jar

http://52.14.168.233:8080/api/v1/salary/search?id=2

sudo apt install redis -y

{ "id": "12345", "name": "John Doe", "salary": 55000.0, "process_date": "2023-10-15", "status": "Active" }

---

attendance

sudo apt update && sudo apt upgrade -y
sudo apt install wget build-essential libncursesw5-dev libssl-dev \
libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.11 -y
sudo ln -s /usr/bin/python3.11 /usr/bin/python
python --version

curl -sSL https://install.python-poetry.org | python -

source ./.bashrc
sudo apt install git -y
git clone https://github.com/OT-MICROSERVICES/attendance-api.git

sudo apt install python3-venv -y

python -m venv myenv --without-pip
source myenv/bin/activate



wget -O- https://repo.liquibase.com/liquibase.asc | gpg --dearmor > liquibase-keyring.gpg && \
cat liquibase-keyring.gpg | sudo tee /usr/share/keyrings/liquibase-keyring.gpg > /dev/null && \
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/liquibase-keyring.gpg] https://repo.liquibase.com stable main' | sudo tee /etc/apt/sources.list.d/liquibase.list

sudo apt-get update
sudo apt-get install liquibase -y

export path in .bashrc

sudo apt update
sudo apt install openjdk-11-jdk -y

export path in .bashrc

sudo apt update
sudo apt install postgresql postgresql-contrib -y
sudo systemctl start postgresql
sudo systemctl enable postgresql
sudo -u postgres psql
ALTER USER postgres PASSWORD 'password';
\du roles
\q quit
\l databases
\dt tables
DROP DATABASE attendance_db;
CREATE DATABASE attendance_db;
GRANT ALL PRIVILEGES ON DATABASE attendance_db TO postgres;

psql -U postgres -h 127.0.0.1 -p 5432

\c attendance_db

INSERT INTO records (id, name, status)
VALUES (1, 'John', 'active');

sudo apt install pylint -y    

sudo pip install peewee voluptuous redis json-logging-py jsonformatter python-json-logger flask-caching prometheus-flask-exporter flasgger pytest-mock pytest pytest-cov Flask psycopg2 PyYAML

sudo apt-get install postgresql -y

pip install flasgger

sudo apt-get install libpq-dev -y

pip3 install psycopg2-binary

sudo apt install postgresql postgresql-contrib -y

sudo systemctl start postgresql

new start

export PATH="/home/ubuntu/.local/bin:$PATH"

sudo apt upgrade -y

remove the above and check why also what happens with ubuntu 22 and ubuntu 20

---

frontend

sudo apt update

sudo apt install git -y

git clone https://github.com/OT-MICROSERVICES/frontend.git

git clone https://github.com/shubhopscloud/frontend (corrected)

cd frontend

mkdir public

cd public

vim index.html

<!DOCTYPE html>

<html lang="en"> <head> <meta charset="utf-8"> <link rel="icon" href="%PUBLIC_URL%/favicon.ico"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Your React App Title</title> </head> <body> <noscript> You need to enable JavaScript to run this app. </noscript> <div id="root"></div> </body> </html>

cd ..

sudo apt install npm -y

npm install

npm install react-scripts

NODE_OPTIONS=--max_old_space_size=4096 npm run build

sudo apt update

sudo apt install nginx -y

sudo mkdir /var/www/frontend

sudo cp -r build/* /var/www/frontend/

sudo cd /etc/nginx/sites-enabled

sudo vim default -> add server ip near server keyword

root /var/www/frontend;

sudo systemctl restart nginx

---

scylla server

DESCRIBE KEYSPACES;

DROP KEYSPACE example_keyspace;

LIST ROLES;

DROP ROLE example_role;

employee_api_role

salary_api_role

ALTER ROLE cassandra WITH PASSWORD = 'password';

CREATE KEYSPACE employee_db WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

CREATE ROLE scylladb WITH PASSWORD = 'password' AND LOGIN = true;

GRANT CREATE ON KEYSPACE employee_data TO scylladb;

GRANT SELECT ON KEYSPACE employee_data TO scylladb;

GRANT MODIFY ON KEYSPACE employee_data TO scylladb;

INSERT INTO salary_db.employee_salary (id, name, salary, process_date, status) VALUES ('12346', 'Shubham Doe', 45000.0, '2024-10-15', 'UnActive');

CREATE TABLE IF NOT EXISTS employeetable ( id text, name text, designation text, department text, joining_date date, address text, office_location text, status text, email text, phone_number text, process_date text, salary float, PRIMARY KEY (id, process_date) ) WITH CLUSTERING ORDER BY (process_date DESC);
inside migration.json

INSERT INTO employee_db.employee_info (id, name, designation, department, joining_date, address, office_location, status, email, phone_number) VALUES ('12346', 'Shubham', 'Software Engineer', 'Engineering', '2023-10-15', '123 Main Street, City, Country', 'Office A', 'Active', 'john.doe@example.com', '+1 (123) 456-7890');

INSERT INTO employee_data.employeetable ( id, name, designation, department, joining_date, address, office_location, status, email, phone_number, process_date, salary ) VALUES ( '12350', 'Shubham Doe', 'Some Designation', 'Some Department', '2024-01-01', '123 Main St, City, Country', 'Office A', 'Active', 'shubham.doe@example.com', '+1 (123) 456-7890', '2023-10-15', 45000.0 );

SELECT id,name, status FROM salary_db.employee_salary ;

SELECT id, name, designation, joining_date, address, office_location, status, phone_number FROM employee_db.employee_info ;

DROP KEYSPACE scylla_db;

LIST ROLES;

changes in api.go, migration files, config.json, migration.json

Load balancer -> 1

/api/v1/employee* - path based routing (through old UI)

/api/v1/salary*

Target groups -> employee, salary, frontend

/api/v1/employee/search/all - healthcheck for target group

/api/v1/salary/search/all

main - stable branch

feature (own names)

prod (release)

---

redis

sudo apt install lsb-release curl gpg -y
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

sudo apt-get update
sudo apt-get install redis -y
redis-cli
redis-server --version

// dasp
echo 'deb http://download.opensuse.org/repositories/home:/cabelo/xUbuntu_22.10/ /' | sudo tee /etc/apt/sources.list.d/home:cabelo.list
curl -fsSL https://download.opensuse.org/repositories/home:cabelo/xUbuntu_22.10/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_cabelo.gpg > /dev/null
sudo apt update
sudo apt install owasp-zap

Run Time Dependency

---

rough work

models/tests/test_user_info.py

models/tests/test_message.py

router/tests/test_cache.py

client/tests/test_postgres_conn.py - 1 fail

client/tests/test_redis_conn.py

- script: | source myenv/bin/activate python3 -m pytest models/tests/test_user_info.py models/tests/test_message.py router/tests/test_cache.py --cov=. --cov-report term-missing --cov-fail-under=70 displayName: 'Run pytest code coverage'

trigger:
- develop

pool:
  name: Default  

steps:
- script: |
    sudo apt update
    sudo apt install pylint -y 
  displayName: 'Install pylint'

- script: |
    pylint --disable=R0903,E0611,E0602,F0010,C0330,C0326,C0304,E0401,C0111,R0801,W0621,W0613,W0612,W0611,C0411,R0201,C0301,C0103,R1705,E1120,R1725,R0124 router/ client/ models/ utils/ app.py
  displayName: 'Run pylint'

- script: |
    sudo apt update
    sudo apt install python3-pip -y
    sudo apt install python3.8-venv
  displayName: 'Install Python3-pip and python3-venv'

- script: |
    python3 -m venv myenv
    source ~/myenv/bin/activate
    cd myagent/workspace/1/s
    python3.8 -m pip install -r requirements.txt
  displayName: 'Install Dependencies'

- script: |
    source ~/myenv/bin/activate
    cd myagent/workspace/1/s
    python3 -m pytest models/tests/test_user_info.py
    python3 -m pytest models/tests/test_message.py
    python3 -m pytest router/tests/test_cache.py
  displayName: 'Run pytest tests'

- script: |
    sudo apt install gunicorn -y
    source ~/myenv/bin/activate
    cd myagent/workspace/1/s
    gunicorn app:app --log-config log.conf -b 0.0.0.0:8080 &
  displayName: 'Run the python API code'

- script: |
    cd
    echo 'deb http://download.opensuse.org/repositories/home:/cabelo/xUbuntu_22.04/ /' | sudo tee /etc/apt/sources.list.d/home:cabelo.list
    curl -fsSL https://download.opensuse.org/repositories/home:cabelo/xUbuntu_22.04/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_cabelo.gpg > /dev/null
    sudo apt update
    sudo apt install owasp-zap -y
  displayName: 'Install Owasp-zap'

- script: |
    sudo apt install default-jre -y
  displayName: 'Install Java 11'

- script: |
    /usr/share/owasp-zap/zap.sh -port 9090 -cmd -quickurl http://3.144.216.246:8080/api/v1/attendance/health/detail > ~/output.txt
  displayName: 'Run a Zap to generate a report'

sonar-scanner -Dsonar.login=1c51834ca7f13ad0c5789e89d537ad24f571ebc1 -Dsonar.projectKey=myproject1 -Dsonar.exclusions="myenv/**/*" -Dsonar.junit.reportPaths=pytest-results.xml -Dsonar.coverage.jacoco.xmlReportPaths=coverage.xml -Dsonar.python.pylint.reportPaths=pylint-report.json

pylint --disable=E0611,E0602,F0010,C0330,C0326,C0304,E0401,C0111,R0801,W0621,W0613,W0612,W0611,C0411,R0201,C0301,C0103,R1705,E1120,R1725,R0124
router/ client/ models/ utils/ app.py --output-format=json > pylint-report.json

coverage run -m pytest models/tests/test_user_info.py models/tests/test_message.py router/tests/test_cache.py

coverage xml -o coverage.xml

pytest --junitxml=pytest-results.xml models/tests/test_user_info.py models/tests/test_message.py router/tests/test_cache.py

---

Postgres

Working

Here we will be setting up the Postgres database server

We will install Postgres on the system by running the below commands which will start the PostgreSQL service as well as enable that service for us so that once we turn on the instances our PostgreSQL service is always up and running!

sudo apt update
sudo apt install postgresql postgresql-contrib -y
sudo systemctl start postgresql
sudo systemctl enable postgresql

We can enter into the Postgres command line shell through the following command

sudo -u postgres psql

<image>

Here, postgres is a default superuser!

Change the password of that user to your respective password, for example, I am giving here password as ‘password’

ALTER USER postgres PASSWORD 'password';

Create a database named attendance_db and grant all permissions to the superuser postgres to perform on the database

CREATE DATABASE attendance_db;
GRANT ALL PRIVILEGES ON DATABASE attendance_db TO postgres;

Select the database using the following command

\c attendance_db

Some handy commands to execute when needed

\du // to get the roles in database
\q // to quit from postgres shell
\l // to list down all the databases 
\dt // to list down tables created in the database
DROP DATABASE attendance_db; // if you want to delete the database

Here as we have set the password and as Postgres is running on port 5432 we would now enter the shell using the following command

psql -U postgres -h 127.0.0.1 -p 5432

You will get a prompt for a password, enter a password and you enter the psql shell successfully again

<image>

We need to edit the pg_hba.conf file to tell which instances are going to forward the request to this Postgres instance

pg_hba.conf

host all all <backend_ip>/32 md5

<image>

postgresql.conf

We have to edit a listener address in this file, by default it is listening on localhost which is 127.0.0.1 but if some other instance wants to access it we need to add that address of the postgres instance by which the other instance is calling the postgres instance, so we can add public IP or private IP according to your use case!

<image>

After configuring the above files, we need to restart the service

sudo systemctl restart postgresql

We dont have any tables inside the database, we will run migrations through liquibase and table is created

liquibase update --driver-properties-file=liquibase.properties

http://3.142.76.6:8080/api/v1/attendance/health

INSERT INTO records (id, name, status, date)
VALUES (1, 'John Doe', 'Active', '2023-10-20');

SELECT id, name, status, date FROM records ORDER BY id DESC

---

In Azure
One simple clarification

NSG -> all connections denied

only AzureLoadBalancer to subnet is allowed

vnet to vnet is allowed

for inbound as well as outbound rules

privatesubnet1 with APIs and frontend

Inbound ->

source section -> \
source ports -> \

destination section -> Any

destination ports -> 22, 80, 8080

Outbound ->

source section -> *

source ports -> *

destination section -> Virtual Network (service tag)

destination ports -> 9042, 6379, 5432

---

privatesubnet2 with APIs and frontend