The deployment of AI systems involves following several patterns and practices to ensure that models perform effectively and reliably in production environments. When designing the AI system architecture, parliament should therefore consider the deployment pattern characteristics discussed below.
Deployment architecture
The deployment architecture of an AI system is determined by two key factors: how the AI algorithm responds to requests, and where the AI model is hosted.
Request-handling patterns:
Batch processing: Data is processed in large batches at scheduled intervals, making this pattern suitable for non-time-sensitive tasks.
- Online serving: Requests are handled in real time as they come in, making this pattern ideal for applications requiring immediate responses.
Streaming: Under this pattern, data streams are continuously processed, enabling near-real-time analysis and predictions.
Hosting location types:
On-premises: Models are deployed on local servers, often for the purpose of enhanced security or to meet specific compliance requirements.
- Cloud: Models are hosted on cloud platforms, offering benefits such as scalability, flexibility and reduced infrastructure management.
- Edge: Models are deployed on edge devices, providing low-latency predictions and offline capabilities, making this approach suitable for Internet of Things (IoT) and mobile applications.
Hybrid: This approach combines on-premises, cloud and edge deployments to optimize performance and resource usage based on specific needs.
The choice of deployment architecture depends on factors such as data sensitivity, response-time requirements, available resources and the specific use case of the AI system.
Scalability
It is important to understand the average number of requests the AI system will receive, along with its life cycle. These factors will determine the deployment scalability characteristics:
Horizontal scaling: Adding more instances of the model server to handle increased load
Vertical scaling: Enhancing the capacity of existing servers (e.g. by adding more memory or faster central processing units (CPUs))
Auto-scaling: Automatically adjusting the number of model instances based on demand
Latency and throughput
When deploying AI systems, two critical performance metrics to consider are latency and throughput:
- Latency refers to the time it takes for the AI model to respond to a request, which is particularly crucial for real-time applications.
- Throughput measures the number of requests the AI model can process per unit of time, which is essential for high-volume applications.
It is important to establish acceptable values for both latency and throughput to ensure that the system meets the specific needs of the application for which it is intended, and that it can handle the expected workload efficiently.
Model management
Effective AI model management is crucial throughout the entire life cycle of an AI system. However, it becomes particularly important once the AI system is put into operation. A well-designed model management strategy should address several key aspects:
- Versioning: This involves keeping track of different versions of the model, ensuring traceability and the ability to roll back if needed. Proper versioning allows teams to manage changes, compare performance across iterations and maintain a clear history of the model’s changes over time.
- Life cycle management: This approach encompasses the tools and processes for deploying, monitoring, updating and, eventually, retiring models. The aim is to ensure that models are properly maintained throughout their operational life, from initial deployment through to eventual replacement.
- A/B testing: This practice involves running multiple versions of a model simultaneously to compare their performance. A/B testing allows teams to make data-driven decisions about which model version performs best in real-world conditions before full deployment.
Monitoring and observability
Performance metrics: Monitoring metrics such as response time, throughput and resource utilization
Drift detection: Identifying when the model’s performance degrades owing to changes in data distribution
Alerting: Setting up alerts for anomalies or performance degradation
Security
Access control: Ensuring that only authorized users and applications can interact with the model
Data privacy: Protecting sensitive data and adhering to regulations (e.g. GDPR)
Model security: Safeguarding models against adversarial attacks and data poisoning
Continuous integration/continuous deployment (CI/CD)
Automation: Automating the deployment process to reduce errors and deployment time
Testing: Including automated testing (unit, integration, regression) in the deployment pipeline
Rollbacks: Providing mechanisms for quickly reverting to previous versions in case of issues
Resource management
Hardware acceleration: Utilizing graphics processing units (GPUs), tensor processing units (TPUs) or other accelerators for improved performance
Resource allocation: Managing resources to optimize cost and performance
Integration with existing systems: Providing APIs for integration with other systems and services
Data pipelines: Integrating with data ingestion and pre-processing pipelines
Feedback loops: Implementing systems to collect feedback from model predictions to improve future performance
Resilience and fault tolerance
Redundancy: Having multiple instances or backups to ensure availability
Failover: Automatically switching to backup systems in case of failure
Retry logic: Implementing mechanisms to handle transient failures
Auditability and explainability
In most cases, audit logs are mandatory for predictions, inputs and system interactions.
In addition to auditing, explainability tools can be used to interpret AI model decisions, thus improving trust and compliance.
