Building Your Own Scalable Video Streaming Server: Part 3 - Data, Deployment & Durability

Welcome back! In Part 1, we looked at the overall structure, and in Part 2, we jumped into the code — setting up our environment and building the main upload and HLS conversion features.

If you missed previous parts, you can find them here:

Part-1: https://dev.to/awalhossain/building-your-own-scalable-video-streaming-server-part-1-the-big-picture-6o5-temp-slug-9072239
Part-2:https://dev.to/awalhossain/building-your-own-scalable-video-streaming-server-part-2-setup-core-implementation-2eg2-temp-slug-984916

Now, in this final part, we’ll work on the key parts needed to make our project stronger and ready to deploy:

1. The API Server's Role: How we manage and store video metadata using MongoDB.
2. Taking Flight: Strategies for deploying our microservices.
3. Handling Hiccups: Making our system resilient with error handling and monitoring.
4. Supercharging: Potential optimizations and scaling considerations.

Let's get this system ready for the real world!

1. The Librarian's Bookshelf: API Server & MongoDB Deep Dive

While the API Gateway handles requests and the Video Conversion service does the heavy lifting, the API Server acts as the central source of truth for our data. Its main jobs are:

• Managing video metadata (titles, descriptions, status, etc.).
• Handling user data (if you have user accounts).
• Providing APIs for fetching video information.
• Keeping the video processing status up-to-date based on events.

MongoDB Schema - The Blueprint:

We use Mongoose to define the structure of our video data in MongoDB. Let's look at the core VideoSchema:

// api-server/src/app/modules/video/video.model.ts
import mongoose, { model } from "mongoose";
import { VIDEO_STATUS, VIDEO_VISIBILITIES } from "./video.constant"; // Import status constants

const VideoSchema = new mongoose.Schema(
  {
    title: { type: String /* required: true */ },
    description: { type: String, default: "" },
    author: { type: mongoose.Schema.Types.ObjectId, required: true, ref: "User" }, // Link to User model
    duration: { type: String }, // e.g., "10:35"
    viewsCount: { type: Number, min: 0, default: 0 },
    // ... other fields like playlistId, language, category ...
    videoLink: { type: String }, // URL to the final HLS master playlist (e.g., ...master.m3u8)
    rawVideoLink: { type: String }, // Link to original uploaded file (optional)
    videoPath: { type: String }, // Temporary local path during processing (might not be needed long-term)
    fileName: { type: String }, // The unique filename used in storage/processing
    originalName: { type: String, required: true }, // User's original filename
    thumbnailUrl: { type: String }, // URL to the generated thumbnail
    history: { type: Array, default: [] }, // Array to track processing steps
    status: { // <-- Crucial for tracking
      type: String,
      enum: Object.values(VIDEO_STATUS),
      default: VIDEO_STATUS.PENDING, // Starts as pending
    },
    visibility: { type: String, enum: Object.values(VIDEO_VISIBILITIES), default: VIDEO_VISIBILITIES.PUBLIC },
    tags: { type: [String], default: [] },
    size: { type: Number }, // Original file size
    videoConversionTime: { type: String }, // How long conversion took
  },
  { timestamps: true } // Adds createdAt and updatedAt fields automatically
);

export const Video = model("Video", VideoSchema);

Key Fields Explained:

• author: Links the video to the user who uploaded it.

• status: Tracks the video's journey. We use constants for this:
  

  // api-server/src/app/modules/video/video.constant.ts
  export const VIDEO_STATUS = {
      PENDING: "Pending",
      PROCESSING: "Processing", // Overall processing started
      UPLOADING_TO_CLOUD: "Uploading to cloud", // (If tracking HLS upload separately)
      SUCCESS: "Success", // Ready for playback
      FAILED: "Failed", // Something went wrong
  } as const; // Using "as const" for stricter typing
  

• history: An array where we can log timestamps and stages (e.g., { status: 'HLS Conversion Complete', timestamp: Date.now() }). This is useful for debugging.

• videoLink: The most important field for playback – the URL to the master HLS playlist (.m3u8) file in cloud storage.

API Server Logic - Responding to Events:

The API Server doesn't just sit there; it actively listens for messages from RabbitMQ, primarily sent by the API Gateway and Video Conversion service.

// api-server/src/app/events/video.events.ts (Conceptual Structure)
import RabbitMQ from '../../shared/rabbitMQ';
import { API_SERVER_EVENTS } from '../../constants/events'; // Consuming events meant for it
import { VideoService } from '../modules/video/video.service'; // The service handling DB operations

const setupVideoEventListeners = async () => {
  // Listen for the initial metadata insertion request from API Gateway
  await RabbitMQ.consume(API_SERVER_EVENTS.INSERT_VIDEO_METADATA_EVENT, async (msg, ack) => {
    try {
      const videoData = JSON.parse(msg.content.toString());
      await VideoService.createVideo(videoData); // Call service to save to DB
      ack();
    } catch (error) { /* Handle error, maybe Nack */ ack(); }
  });

  // Listen for updates from Video Conversion Service (e.g., duration, status)
  await RabbitMQ.consume(API_SERVER_EVENTS.UPDATE_METADATA_EVENT, async (msg, ack) => {
     try {
        const { id, ...updateData } = JSON.parse(msg.content.toString());
        // Could also update status to PROCESSING here
        await VideoService.updateVideo(id, updateData);
        ack();
     } catch (error) { /* ... */ ack(); }
  });

  // Listen for thumbnail generation completion
   await RabbitMQ.consume(API_SERVER_EVENTS.VIDEO_THUMBNAIL_GENERATED_EVENT, async (msg, ack) => {
     try {
        const { id, thumbnailUrl } = JSON.parse(msg.content.toString());
        await VideoService.updateVideo(id, { thumbnailUrl, /* Update history maybe */ });
        ack();
     } catch (error) { /* ... */ ack(); }
  });

   // Listen for final HLS conversion completion
   await RabbitMQ.consume(API_SERVER_EVENTS.VIDEO_HLS_CONVERTED_EVENT, async (msg, ack) => {
     try {
        const { id, videoLink } = JSON.parse(msg.content.toString());
        // Update status to SUCCESS and add the final HLS link!
        await VideoService.updateVideo(id, { status: 'Success', videoLink });
        ack();
     } catch (error) { /* ... */ ack(); }
  });


  // ... potentially listen for failure events ...
};

The VideoService (video.service.ts) contains the actual Mongoose queries (Video.create(), Video.findByIdAndUpdate(), etc.) to interact with the database.

2. Taking Flight: Deployment Strategies

Running docker-compose up on your machine is great for development, but how do you get this running reliably for others to use? Deployment!

Docker is Key:

We've already created Dockerfiles for each service. These are the blueprints for building container images – standardized packages containing our code and dependencies.

# Example: video-conversion/Dockerfile (Multi-stage build)
# Stage 1: Build the application
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build # Compile TypeScript to JavaScript in /dist

# Stage 2: Create the final, smaller image
FROM node:18-alpine
WORKDIR /app
# Install FFmpeg directly in the final image!
RUN apk update && apk add --no-cache ffmpeg
COPY package*.json ./
# Only install production dependencies
RUN yarn install --production --frozen-lockfile
# Copy built code from the builder stage
COPY --from=builder /app/dist ./dist
# Copy entrypoint script if needed
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh

ENV NODE_ENV=production
EXPOSE 8002 # Expose the port the service runs on
# Command to run when the container starts
CMD ["node", "dist/server.js"]
# Or use entrypoint script: ENTRYPOINT ["./entrypoint.sh"]

Now, let's look at how to run these containers in a production-like environment.

Option 1: Docker Compose (Single Server)

• Concept: Run all containers (services, DB, Redis, RabbitMQ) on a single virtual machine using the docker-compose.yml file we used for development.
• Pros: Simplest approach, good for small projects or internal tools.
• Cons: Doesn't scale easily beyond one machine, single point of failure (if the machine goes down, everything goes down).

Option 2: Docker Swarm (Basic Orchestration)

• Concept: Docker's built-in orchestrator. You can join multiple machines (nodes) into a "swarm" and deploy services across them.
• How: Initialize a swarm (docker swarm init), join worker nodes (docker swarm join), and deploy your docker-compose.yml as a "stack" (docker stack deploy -c docker-compose.yml myapp). Swarm handles basic load balancing and service discovery.
• Pros: Built into Docker, relatively easy transition from Compose, provides redundancy if a node fails (if services have replicas).
• Cons: Less feature-rich and less widely adopted than Kubernetes.

Option 3: Kubernetes (K8s - Advanced Orchestration)

• Concept: The industry standard for managing containerized applications at scale. It's powerful but more complex.
• How: You define your application using Kubernetes objects (written in YAML):
  • Deployment: Manages replicas (pods) of your application, handles rolling updates.
  • Service: Provides a stable IP address and DNS name for accessing your deployments.
  • Ingress: Manages external access (HTTP/S routing) to your services, often handling SSL termination.
  • ConfigMap/Secret: Manage configuration and sensitive data separately from your container images.
  • PersistentVolume/PersistentVolumeClaim: For stateful data like MongoDB.
  • You likely have basic manifests in the k8s/ directories within each service. You'd apply these using kubectl apply -f .
• Pros: Highly scalable, resilient (auto-healing, rescheduling), rolling updates, large community and ecosystem, standard across cloud providers (like AWS EKS, Google GKE, Azure AKS).
• Cons: Steeper learning curve, more complex setup and management.

Choosing Your Path:

• Starting out / Small project? Docker Compose on a single VM is fine.
• Need basic scaling and redundancy? Docker Swarm might be sufficient.
• Building for large scale, high availability, and adopting cloud-native practices? Kubernetes is the way to go, despite the complexity.

3. Handling Hiccups: Error Handling & Monitoring

Things will go wrong in production. Being prepared is key.

Error Handling:

• Catching Async Errors: We use utilities like catchAsync in controllers to avoid repetitive try...catch blocks around async route handlers.
  

  // api-gateway/src/shared/catchAsyncError.ts
  import { Request, Response, NextFunction, RequestHandler } from 'express';
  const catchAsync = (fn: RequestHandler) => {
    return (req: Request, res: Response, next: NextFunction) => {
      Promise.resolve(fn(req, res, next)).catch((err) => next(err));
    };
  };
  export default catchAsync;
  

• Global Error Handler: Each service's app.ts should have a final middleware that catches any unhandled errors passed via next(err). This handler standardizes error responses.

• Specific Errors: try...catch blocks are still essential for critical operations like file system access, database calls, or FFmpeg execution (hlsConvertProcessor.ts has .on('error', ...)).

• Queue Failures: BullMQ workers have .on('failed', ...) listeners (jobWorker.ts). Log these errors! You might need strategies to automatically retry certain failures or flag them for manual review.

Monitoring:

You can't fix what you can't see!

• Logging: We're using Winston (shared/logger.ts) for structured logging (JSON format is best for machines to parse). winston-daily-rotate-file helps manage log file size. Ensure you log meaningful information, including request IDs to trace requests across services.
• Application Performance Monitoring (APM): Tools like Sentry (which you have integrated in video-conversion/src/app.ts) are invaluable. They automatically capture unhandled exceptions, performance bottlenecks, and provide rich context (stack traces, request data) to help you debug faster.
• Infrastructure Metrics: Monitor the health of your containers and nodes (CPU, memory, disk I/O, network). Cloud providers offer built-in monitoring, or you can use tools like Prometheus and Grafana.
• Queue/Broker Monitoring: Keep an eye on RabbitMQ (the management UI on port 15672 is helpful) and Redis/BullMQ to spot backed-up queues or high error rates.

4. Supercharging Your Server: Optimizations & Scaling

As your platform grows, you'll need to optimize.

• Video Conversion Service:
  • Scale Horizontally: This service is CPU-bound due to FFmpeg. Run multiple instances (replicas/pods) behind a load balancer (handled by Swarm/K8s Services). BullMQ naturally distributes jobs across available workers.
  • Hardware: Consider using VMs/nodes optimized for CPU performance. GPU acceleration for FFmpeg is possible but adds complexity.
  • Concurrency: Fine-tune the BullMQ worker concurrency (jobWorker.ts) based on your server resources. Too high can overload the server; too low underutilizes it.
• Database (MongoDB):
  • Indexing: Ensure you have indexes on fields used frequently in queries (e.g., author, status, createdAt). Use MongoDB's explain() to analyze query performance.
  • Read Replicas: If you have many users browsing videos (read-heavy), set up MongoDB read replicas to distribute the load.
• API Services (Gateway/Server):
  • Statelessness: These are generally stateless (don't store session data in memory), making them easy to scale horizontally by adding more replicas.
  • Caching: Use Redis (which you already have for BullMQ!) to cache frequently accessed, non-critical data – user sessions, results of expensive queries, maybe even pre-signed URL results briefly.
• Content Delivery Network (CDN):
  • Crucial for Playback: Put a CDN (like AWS CloudFront, Cloudflare, Akamai) in front of your cloud storage bucket (DO Spaces/S3).
  • Benefits: The CDN caches the HLS playlist (.m3u8) and video segment (.ts) files in edge locations closer to your users worldwide. This dramatically improves video start times, reduces buffering, and lowers your cloud storage egress costs.

5. Conclusion: Where to Go From Here?

And that's a wrap on our three-part journey! We've gone from a high-level architectural idea (Part 1), through the nitty-gritty code implementation (Part 2), to managing data, deploying, and ensuring robustness (Part 3).

You now have a solid foundation for building a scalable video streaming server. The specific implementation details might evolve, but the core concepts – microservices, direct uploads, asynchronous processing queues, HLS conversion, and monitoring – are fundamental.

Next Steps:

• Explore the Code: Clone the repository (if available) and experiment.
• Deploy It: Try deploying using Docker Compose or tackle Kubernetes.
• Contribute/Improve: Identify areas for enhancement.

Future Features to Consider:

• Live Streaming (Requires different FFmpeg setup and protocols like RTMP/WebRTC)
• Video Playlists
• User Comments & Likes
• Detailed Analytics
• Digital Rights Management (DRM) for content protection
• Admin Interface

Building complex systems is challenging but incredibly rewarding. Hopefully, this series has demystified the process and equipped you to build your own amazing video platform. Happy coding!

Explore the full implementation and grab the code from the GitHub repository:

Github-Server: https://github.com/AwalHossain/video-streaming-server
Github-Client: https://github.com/awalHossain/video-streaming-client

Do follow me on my Socials to stay updated on tech stuff: