DZone Spotlight

Today, in modern backends, you probably have those distributed job queues for everything, including sending emails, processing payments, generating reports, and syncing data to third parties. As soon as you add retries to handle transient failures, however, you inherit a hard problem: how do you ensure that when the network, worker, or broker can fail at any point, your job runs exactly once? The short answer is: "exactly once delivery" is a great concept, but in practice it's mostly fiction given the nature of distributed systems. What you really can make is at-least-once delivery + idempotent processing, yielding exactly once effects. This article demonstrates how to accomplish this in Node.js with a tangible, functioning implementation. The Problem: Retries Cause Duplicates Take a worker that charges the customer and then marks the job completed TypeScript async function processJob(job) { await chargeCustomer(job.customerId, job.amount); await markJobComplete(job.id); } This seems fine until you consider that the work crashes after chargeCustomer succeeds but before markJobComplete executes. Because the queue does not receive an acknowledgement, it redelivers the job. The customer gets charged twice. This is not a rare edge case. Do any significant amount of throughput and workers fall over, containers reschedule, network calls default after the server has already worked its way through them. If you have side effects in your job, then you can always assume any job may be delivered more than once. The Solution: Idempotency Keys The main concept is to give every job created a unique, deterministic idempotency key and log the output of processing that key. The worker only checks if a key has been processed before doing any work. If so, it simply returns the result that was saved and does not redo the work. Here is the schema for how we can keep track of processed jobs. TypeScript CREATE TABLE processed_jobs ( idempotency_key VARCHAR(255) PRIMARY KEY, status VARCHAR(20) NOT NULL, -- 'in_progress' | 'completed' result JSONB, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), completed_at TIMESTAMPTZ ); The job must encode its sensitive payload and key, not randomly generated at enqueue time. Good keys will be things like charge:order_12345, which will hopefully be stable across retries of the same logical operation. A Working Implementation The trick is to acquire the key atomically before doing anything useful. To claim the job, we execute a single atomic operation in PostgreSQL, which is an INSERT... ON CONFLICT DO NOTHING: TypeScript const { Pool } = require('pg'); const pool = new Pool(); async function processIdempotent(idempotencyKey, work) { const client = await pool.connect(); try { // Step 1: Try to claim the key atomically. const claim = await client.query( `INSERT INTO processed_jobs (idempotency_key, status) VALUES ($1, 'in_progress') ON CONFLICT (idempotency_key) DO NOTHING RETURNING idempotency_key`, [idempotencyKey] ); // Step 2: If we did NOT claim it, someone else already did. if (claim.rowCount === 0) { const existing = await client.query( `SELECT status, result FROM processed_jobs WHERE idempotency_key = $1`, [idempotencyKey] ); const row = existing.rows[0]; if (row.status === 'completed') { return row.result; // Return the cached result — no double work. } // Still in progress elsewhere — let the queue retry later. throw new Error('JOB_IN_PROGRESS'); } // Step 3: We own the key. Do the actual work. const result = await work(); // Step 4: Record the result. await client.query( `UPDATE processed_jobs SET status = 'completed', result = $2, completed_at = now() WHERE idempotency_key = $1`, [idempotencyKey, result] ); return result; } finally { client.release(); } } Now the worker becomes: TypeScript async function processJob(job) { return processIdempotent(`charge:${job.orderId}`, async () => { const charge = await chargeCustomer(job.customerId, job.amount); return { chargeId: charge.id }; }); } The key is already completed, and whenever this job is delivered the second (or more) time it will return the chargeId that was previously stored without charging again. Handling the Stuck "in_progress" Case The last failure mode that remains is where a worker picks a key, sets it to in_progress, and then dies without completing. Now this key is stuck, and any retry gives JOB_IN_PROGRESS forever. The solution is an expiration-lease for the lease. 1. Add locked_until column, make expired lock reclaimable: TypeScript const claim = await client.query( `INSERT INTO processed_jobs (idempotency_key, status, locked_until) VALUES ($1, 'in_progress', now() + interval '5 minutes') ON CONFLICT (idempotency_key) DO UPDATE SET locked_until = now() + interval '5 minutes', status = 'in_progress' WHERE processed_jobs.status = 'in_progress' AND processed_jobs.locked_until < now() RETURNING idempotency_key`, [idempotencyKey] ); It only requires a lock if the circuit is in progress and its lease has timed out, which means that some worker abandoned it earlier. The completed jobs will never be reclaimed, because the WHERE excludes them. Why not simply use a distributed lock One of the most common instincts here is to grab ourselves a Redis lock (if not using redis-lock, do SETNX with a TTL). Despite a lock being a solution for mutual exclusion, they do not solve idempotency by themselves. The job is already done, but because it uses a lock to prevent two workers from running at once. If you only use a lock, the job will be reprocessed when the lock expires and a redelivery is attempted. What you need is a permanent record of completion and that is what the processed jobs table provides. Locks and idempotency keys address two separate problems, yet durable systems typically require both. Takeaways Assume at-least-once delivery; make each job handler idempotent.Use the intent from job to derive idempotency keys; ensure they are stable across retries.Store results and claim keys atomically with INSERT... ON CONFLICT so duplicates return the cached result.Lease with an expiration because crashed workers should not block a key forever. Idempotency is certainly not useful, but it helps the queue to be the difference between something you can trust and a facility that will silently double charge your customers because of load. Treat it as a first-class citizen, because adding it via retrofitting after failing is way worse.

RabbitMQ is an enterprise-grade open-source messaging and streaming broker. In this blog, you will learn some basic concepts of RabbitMQ and how to use it in a Spring Boot application. Enjoy! Introduction Before diving into the programmatic details, first some concepts need to be explained. Do realize that in this blog, only the surface is scratched from what is possible with RabbitMQ. A detailed overview can be found in the official RabbitMQ documentation. Several protocols are supported by RabbitMQ. In this blog, the AMQP 0-9-1 protocol will be used. AMQP stands for Advanced Message Queuing Protocol. RabbitMQ receives messages from a publisher, a producing application, and routes them to consumers, applications that process the messages. A publisher publishes messages to an exchange (like a mailbox). The exchange then routes the messages to queues using bindings. RabbitMQ then delivers the messages to the consumers who are subscribed to the queues. The process is shown in the figure below. In the examples in the remainder of this blog, you will make use of a Topic Exchange. There are different exchange types, but for the sake of simplicity, only one will be used. A topic exchange routes messages to one or many queues, based on a message routing key. Topic exchanges are commonly used for multicast routing of messages. Sources used in this blog are available on GitHub in module topics. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose. Create Spring Boot Application In order to get started, you navigate to the Spring Initializr and add the following dependencies: Spring Web: in order to be able to send messages via an http request.Docker Compose Support: in order to start a RabbitMQ container when the application starts.Spring for RabbitMQ: in order to integrate Spring Boot with RabbitMQ. You will build the following: One Exchange with one Topic.Publish a general message to the topic which will be consumed by consumer A and consumer B.Publish a specific message to the topic which will be only consumed by consumer B. In order to send a general and a specific message, two HTTP endpoints are created in the MessageController. Java @RestController public class MessageController { private MessageService messageService; public MessageController(MessageService messageService) { this.messageService = messageService; } @RequestMapping( method = RequestMethod.POST, value = "send-general" ) public ResponseEntity<Void> sendGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message", message); return new ResponseEntity<>(HttpStatus.CREATED); } @RequestMapping( method = RequestMethod.POST, value = "send-specific" ) public ResponseEntity<Void> sendSpecificMessage(@RequestBody String message) { messageService.sendMessage("event.specific.message", message); return new ResponseEntity<>(HttpStatus.CREATED); } } The requests are forwarded to a MessageService.sendMessage method, which takes a routingKey and the message as arguments. The message is taken from the http request body, the routingKey is hardcoded. Remember that the routingKey determines to which queue the message will be routed. In the service, you make use of Spring Boot's RabbitTemplate in order to send the message to RabbitMQ. Java @Service public class MessageService { private RabbitTemplate rabbitTemplate; public MessageService(RabbitTemplate rabbitTemplate) { this.rabbitTemplate = rabbitTemplate; } public void sendMessage(String routingKey, String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.TOPIC_EXCHANGE_NAME, routingKey, message); } } Bind Consumer A Consumer A will consume general messages. The queue needs to be bound to the Topic Exchange with the routing key. Create a RabbitMqConfig class with: A TopicExchange bean with name events.exchange.A Queue bean for consumer A with name consumer-a.queue.A binding bean for consumer A connecting the queue of consumer A to the TopicExchange with the routing key for the general messages. Do note that the name of the queue in method bindingConsumerA needs to match the queueConsumerA bean name. Java Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String TOPIC_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_GENERAL_MESSAGE = "event.general.*"; @Bean TopicExchange eventsExchange() { return new TopicExchange(TOPIC_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_GENERAL_MESSAGE); } } Create Consumer A Next thing to do is to consume the messages from queue A. Create a Component named ReceiverA. Annotate the method for processing the messages with @RabbitListener and connect it to queue A. When receiving the message, just print it to the console. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } Run the Application In order to run the application, you will need RabbitMQ. Since you have added Docker Compose Support to the project earlier, you can just add a compose.yaml in the root of the repository. YAML services: rabbitmq: image: rabbitmq:3.13-management-alpine # Stable, lightweight, includes management UI container_name: rabbitmq ports: - "5672:5672" # AMQP - "15672:15672" # Management console environment: RABBITMQ_DEFAULT_USER: secret RABBITMQ_DEFAULT_PASS: myuser Also add the connection parameters for RabbitMQ to the application.properties file. Properties files spring.rabbitmq.host=localhost spring.rabbitmq.port=5672 spring.rabbitmq.username=secret spring.rabbitmq.password=myuser Start the application. Shell mvn spring-boot:run You will notice that RabbitMQ is started automatically. Send a general message. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" The console log will print the following. Plain Text Queue Consumer A received <This is a general message> Stop the application. Bind Consumer B Consumer B will process general messages, but also specific messages. Add to the RabbitMqConfig the queue for consumer B, and bind it to the exchange with respectively the general message routing key and the specific message routing key. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String QUEUE_CONSUMER_B = "consumer-b.queue"; public static final String TOPIC_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_GENERAL_MESSAGE = "event.general.*"; public static final String ROUTING_KEY_SPECIFIC_MESSAGE = "event.specific.*"; ... @Bean public Queue queueConsumerB() { return new Queue(QUEUE_CONSUMER_B, false); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_GENERAL_MESSAGE); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } } Create Consumer B Consumer B is created just like consumer A. Create a ReceiverB class in order to receive the queue B messages. Java @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_B) public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Run the Application Start the application. Shell mvn spring-boot:run Send a general message. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" The message is now received by Consumer A and Consumer B. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Send a specific message. Shell curl -X POST http://localhost:8080/send-specific \ -H "Content-Type: text/plain" \ -d "This is a specific message" The message is only received by Consumer B. Plain Text Queue Consumer B received <This is a specific message> Management Console Also take a look at the RabbitMQ management console, which is accessible at http://localhost:15672/. Here you can see the exchanges, the queues, the bindings, etc. Conclusion In this blog, you learned some basics of RabbitMQ using the AMQP 0-9-1 protocol. You learned how easy it is to integrate this within your Spring Boot application.

By Gunter Rotsaert CORE

Somewhere right now, an engineer is making the case to rewrite a working PHP app in Node, and the pitch includes the word "modern." I have heard a version of this for fifteen years. The app ships. The customers are happy. The code is unfashionable. And somebody wants to tear it down and rebuild it on a stack that looks better on a resume. I have shipped software for more than 20 years, and these days I spend a lot of my time watching AI coding agents write it. So here is a take that is going to sound backward: the thing everyone makes fun of PHP and Laravel for — that they are rigid, opinionated, and boring- is the exact thing that makes coding agents so good at them. When a machine writes a big chunk of your code, the most valuable thing your framework can give you is predictability, not flexibility. And the trendy, flexible stack the rewrite crowd wants is quietly making your AI tooling worse. The Thing That Makes a Stack Feel Modern Makes AI Worse at It A coding agent is a pattern matcher with a context window. It is good at your codebase to the degree that your codebase looks like the millions of others it trained on, and to the degree that it can guess where things go without reading the whole repo first. A bespoke Node service is the opposite of that. Node and Express enforce almost no structure, and that gets sold as a feature. You arrange the project however your team likes. One team puts routes in routes/. Another co-locates them with handlers. A third invents a domain-folder layout from a blog post someone read once. Controllers, services, models, and middleware live wherever this particular team decided. For a senior team, that freedom is genuinely nice. It is also poison for an agent. When you ask the model to add an endpoint, it first has to infer your project's private conventions from whatever it can see, then guess at the rest. Two runs of the same prompt come out different, because there is no canonical answer to "where does this go." The agent burns its effort rebuilding context your layout never standardized, instead of writing the feature. This is not really a Node problem. It is a configuration-over-convention problem, and it shows up anywhere the layout is a per-team decision. Even Django, a real framework with real conventions, leaves you enough rope (models in one file or split across many, your pick of API layer) that the AI output wobbles more than it does in a stricter framework. The more the framework leaves up to you, the more the agent has to guess. Convention Over Configuration Was an AI Strategy Before There Was AI Now open any Laravel project, built by any team, in any country. You already know where everything is. Models in app/Models. Controllers in app/Http/Controllers. Policies in app/Policies. Migrations follow the same timestamped naming every time. This is convention over configuration, the principle Rails made famous, and Laravel built its whole developer experience around. For two decades it was sold as a way to stop bikeshedding and onboard humans faster. It turns out it was an AI strategy the whole time, and nobody knew it yet. When the file always lives in the same place, and the code always follows the same idiom, the model has effectively seen your project a million times before it ever touches it. The structure it is predicting is not your team's private invention. It is the global standard, which is exactly what the model trained on. So the generated code comes out idiomatic, lands in the right directory, and looks the same across two runs of the same prompt. Laravel even ships official AI-assisted-development docs now, plus a tool called Boost that feeds an agent the framework's own conventions. That is the tell. The thing that makes a framework easy for a new human to read — everything is where you would expect — is the same thing that makes it easy for a machine. AI just raised the payoff on being predictable. What This Looks Like When You Actually Ship I am not making this argument in the abstract. I am watching it play out in my own company's products. Our newest product, ProductWave, is built entirely on PHP and Laravel. Not out of nostalgia. We got tired of the JavaScript churn, the dependency hell, the new framework every nine months, the constant re-platforming. Laravel is opinionated in the right places. You get auth, queues, an ORM, scheduling, and a sane directory structure on day one, so you stop arguing with the tooling and start shipping features. The AI part is what made the bet pay off harder than I expected. Because Laravel's conventions are so consistent, the agents we use write noticeably better code in our Laravel apps than in a from-scratch Node service where every team invented its own layout. Same file, same place, every time. So the output is idiomatic instead of improvised, and it holds up across runs. Here is the difference in the terms that actually matter when an agent is writing your code: What the coding agent facesConvention stack (Laravel, Rails)Bespoke stack (hand-rolled Node)Where a new controller goesSame path in every project on earthWherever this team decided, if anyone didStyle of the generated codeMatches the public examples it trained onMatches your house pattern, if one existsTwo runs of the same promptMostly consistentVary run to runContext it must rebuild per repoAlmost none, the structure is the standardMost of it, the layout is privateHow a new engineer (or agent) reads itLike every other projectLike a new language None of this needs the framework to be technically better on every axis. It needs the framework to make the same decision every time, so neither your new hire nor your AI has to wonder. PHP Got Written Off Years Ago. It Is Worth a Second Look. I know the objection, because the rewrite pitch always carries it: PHP is slow, untyped, stuck in 2010. If your last serious PHP experience was a PHP 5.6 codebase, that picture is more than a decade out of date. PHP 8 added a JIT compiler and a real type system. Union types, readonly properties, enums, the match expression, and Fibers for async are all standard now: PHP // PHP 5.6 function process($value) { if (is_int($value) || is_float($value)) { return calculate($value); } } // PHP 8.x function process(int|float $value): float { return calculate($value); } The performance cliche is just as stale. When Tumblr moved its fleet from PHP 5 to PHP 7, the engineering team documented latency dropping by half and CPU load falling at least 50 percent, and PHP 8 kept climbing from there. This is not a dead language. By W3Techs' numbers, it still runs roughly three-quarters of the websites with a known server-side language, and it powers production at the scale of Etsy and Slack. There are good, boring reasons companies still run on PHP. It is unfashionable on Hacker News, which is a very different thing from being dead. The Rewrite Reflex Gets It Backward So why does the rewrite argument keep coming up? Usually it is what I call resume-driven development. The stated reason is "PHP is outdated." The real reason is that an engineer wants the trendy stack on their resume for the next interview. That is rational for the individual and a disaster for the roadmap. I say that as someone who has approved the rewrite and regretted it! Every team I have watched hit this fork landed the same way. The ones that worked said no to the rewrite, modernized the stack they had, and kept shipping customer value. The ones that did not approve it, spent the better part of two years rebuilding what already worked, shipped nothing new in the meantime, and watched competitors eat their lunch. The AI era adds a line to that math the rewrite crowd never accounts for. When you tear down a legible, convention-driven Laravel app and rebuild it as a bespoke service in a flexible stack, you are not just paying the old rewrite tax. You are actively making your codebase harder for the AI tooling you are betting your future speed on. You are trading a structure the model understands for one it has to relearn. You are spending two years to make your own agents worse at their job. That is the opposite of modernization. What You Should Actually Do You do not have to adopt PHP to use any of this. The principle is about convention, not about a language. For greenfield work, bias toward an opinionated framework. Laravel, Rails, and the convention-heavy frameworks in any language give an agent a predictable surface to generate against. The "we will assemble our own stack" instinct feels powerful and quietly costs you AI quality.Modernize the app you have instead of rewriting it. If you are on an old PHP or Laravel version, upgrade it and adopt the conventions fully. You will get more out of your agents from a current, consistent codebase than from a brand-new language, at a fraction of the cost and risk.If you are stuck in a flexible stack, impose convention anyway. Pick a canonical layout, document it, lint for it, and keep it identical across services. The agent cannot read your mind, but it will follow a structure you actually enforce. Most of the AI-quality gap closes the moment the layout stops being a per-team decision.Stop treating "boring" as an insult. Boring means predictable. Predictable means staffable, and now it means legible to a machine too. In an AI shop, that is the competitive choice, not the compromise. The Bottom Line For fifteen years, the knock on Laravel was that it makes your decisions for you. That was always a strange thing to complain about. Now it is the entire advantage, and the agents are the ones cashing it in.

Face recognition has become one of the most widely used applications of artificial intelligence and computer vision. From smartphone authentication and smart surveillance systems to attendance management and access control solutions, facial recognition technology plays an important role in identifying individuals automatically. Advances in machine learning and image processing have made it possible to develop accurate face recognition systems using open-source tools and libraries. This project demonstrates the implementation of a real-time face recognition application using Python, OpenCV, Dlib, and the Face Recognition library. The application captures video input, detects human faces, generates facial feature encodings, and compares them against a database of known individuals. Once a match is identified, the system displays the corresponding name on the video frame. The project serves as an excellent example of practical computer vision implementation and provides a foundation for developing more advanced AI-based recognition systems. Environment Setup Before executing the application, it is important to prepare the development environment properly. Since the Face Recognition library depends heavily on Dlib, several prerequisites must be installed. The first requirement is Python. Any recent Python version can be used, and users who already have Anaconda installed can use the existing Python environment. The next step involves installing CMake. CMake is required to build and compile Dlib successfully. After downloading the latest stable Windows installer, the installation process should include adding CMake to the system PATH. Once installation is completed, the system should be restarted to ensure all environment variables are updated correctly. Another important requirement is Microsoft Visual Studio Build Tools. During installation, the "Desktop Development with C++" workload must be selected. These build tools provide the necessary compiler and development libraries required by Dlib. After installation, a system restart is recommended. Once the system prerequisites are installed, the Python libraries can be installed using the following commands: Python pip install cmake pip install opencv-python pip install dlib pip install face-recognition In some cases, Dlib installation may fail because of incomplete dependencies. Running the installation through an Anaconda Prompt with administrator privileges often resolves such issues. After installation, Dlib can be verified by importing it into Python and printing the installed version. Python #importing the required libraries import cv2 import face_recognition #capture the video from default camera webcam_video_stream = cv2.VideoCapture('images/testing/image1.mp4') #load the sample images and get the 128 face embeddings from them image1_image = face_recognition.load_image_file('images/samples/image1.jpg') image1_face_encodings = face_recognition.face_encodings(image1_image)[0] image2_image = face_recognition.load_image_file('images/samples/image2.jpg') image2_face_encodings = face_recognition.face_encodings(image2_image)[0] sen_image = face_recognition.load_image_file('images/samples/sen.jpg') sen_face_encodings = face_recognition.face_encodings(sen_image)[0] #save the encodings and the corresponding labels in seperate arrays in the same order known_face_encodings = [image1_face_encodings, image2_face_encodings, sen_face_encodings] known_face_names = ["Person1", "Person2", "Person3"] #initialize the array variable to hold all face locations, encodings and names all_face_locations = [] all_face_encodings = [] all_face_names = [] #loop through every frame in the video while True: #get the current frame from the video stream as an image ret,current_frame = webcam_video_stream.read() #resize the current frame to 1/4 size to proces faster current_frame_small = cv2.resize(current_frame,(0,0),fx=0.25,fy=0.25) #detect all faces in the image #arguments are image,no_of_times_to_upsample, model all_face_locations = face_recognition.face_locations(current_frame_small,number_of_times_to_upsample=1,model='hog') #detect face encodings for all the faces detected all_face_encodings = face_recognition.face_encodings(current_frame_small,all_face_locations) #looping through the face locations and the face embeddings for current_face_location,current_face_encoding in zip(all_face_locations,all_face_encodings): #splitting the tuple to get the four position values of current face top_pos,right_pos,bottom_pos,left_pos = current_face_location #change the position maginitude to fit the actual size video frame top_pos = top_pos*4 right_pos = right_pos*4 bottom_pos = bottom_pos*4 left_pos = left_pos*4 #find all the matches and get the list of matches all_matches = face_recognition.compare_faces(known_face_encodings, current_face_encoding) #string to hold the label name_of_person = 'Unknown face' #check if the all_matches have at least one item #if yes, get the index number of face that is located in the first index of all_matches #get the name corresponding to the index number and save it in name_of_person if True in all_matches: first_match_index = all_matches.index(True) name_of_person = known_face_names[first_match_index] #draw rectangle around the face cv2.rectangle(current_frame,(left_pos,top_pos),(right_pos,bottom_pos),(255,0,0),2) #display the name as text in the image font = cv2.FONT_HERSHEY_DUPLEX cv2.putText(current_frame, name_of_person, (left_pos,bottom_pos), font, 0.5, (255,255,255),1) #display the video cv2.imshow("Webcam Video",current_frame) if cv2.waitKey(1) & 0xFF == ord('q'): break #release the stream and cam #close all opencv windows open webcam_video_stream.release() cv2.destroyAllWindows() Project Overview The objective of this project is to identify known individuals appearing in a video stream. The system uses previously stored facial images as references. Each reference image is converted into a mathematical representation known as a facial embedding. When a video frame is processed, faces are detected and converted into similar embeddings. These embeddings are compared with stored embeddings, and if a match is found, the person's name is displayed on the screen. The workflow consists of the following stages: Load reference imagesGenerate face encodingsCapture video framesDetect faces in each frameGenerate facial embeddingsCompare embeddingsDisplay recognition results Loading Sample Images The application begins by loading images of known individuals. These images act as training references for the recognition process. The Face Recognition library provides a convenient function called load_image_file() which reads image files and converts them into arrays suitable for processing. For each image, the system generates a 128-dimensional face encoding. These encodings capture unique facial characteristics and serve as the identity signature of an individual. The generated encodings are stored in an array together with corresponding labels. Maintaining the same order between encodings and names ensures accurate identification later during matching. Video Stream Processing After loading the known faces, the application opens a video source. The video can be captured from a webcam or from a video file. Each frame is processed continuously within a loop. Since facial recognition is computationally intensive, the frame is resized to one-quarter of its original size. This optimization significantly improves processing speed while maintaining sufficient accuracy. Reducing image size helps the system achieve smoother real-time performance, especially on systems without dedicated graphics hardware. Face Detection Once a frame is resized, the system searches for human faces within the image. The Face Recognition library internally uses Dlib's face detector to locate faces. The code specifies the HOG (Histogram of Oriented Gradients) model, which is known for its balance between accuracy and performance. The detector returns coordinates representing the location of each detected face. These coordinates include the top, right, bottom, and left boundaries of the face. Because detection is performed on a reduced-size image, the coordinates are multiplied by four to map them back to the original frame dimensions. Face Encoding Generation After face locations are identified, the application generates face encodings for each detected face. A face encoding is a numerical representation consisting of 128 values. These values describe the unique characteristics of a person's face and allow efficient comparison between different faces. The encoding process is one of the most important stages of the recognition pipeline because it transforms visual information into mathematical data suitable for machine learning comparison. Face Matching The generated encoding from the current frame is compared against the stored encodings of known individuals. The compare_faces() function performs this comparison and returns a list indicating whether each stored encoding matches the current face. If a match exists, the application retrieves the index of the matching encoding and uses that index to obtain the corresponding person's name. If no match is found, the face is labeled as "Unknown Face." This approach provides a simple yet effective mechanism for identifying individuals in real time. Displaying Recognition Results Once a face is identified, the application visually highlights the result. A rectangle is drawn around the detected face using OpenCV. The recognized person's name is displayed near the face boundary using text rendering functions. These visual annotations provide immediate feedback and allow users to observe recognition results directly within the video stream. The video continues processing until the user presses the "Q" key to terminate execution. Applications The concepts demonstrated in this project can be applied to numerous real-world scenarios. Common applications include: Employee attendance systemsSmart access controlSecurity monitoringVisitor identificationAutomated authenticationEducational research projectsAI-powered surveillance systems Organizations can integrate similar systems into existing infrastructure to improve security and operational efficiency. Future Enhancements Although the current implementation performs effectively, several improvements can be introduced. Future enhancements may include: GPU acceleration for faster processingDeep learning-based face detection modelsMulti-camera supportCloud-based facial databasesEmotion detectionFace mask recognitionAttendance report generationIntegration with mobile applications These improvements would increase scalability and accuracy while enabling deployment in larger environments. Conclusion This project demonstrates the practical implementation of real-time face recognition using Python, OpenCV, Dlib, and the Face Recognition library. By combining face detection, feature extraction, and facial matching techniques, the system successfully identifies known individuals appearing in a video stream. The installation process, while requiring several dependencies such as CMake and Visual Studio Build Tools, provides a stable environment for Dlib and facial recognition functionality. The project highlights how modern computer vision libraries can be used to build intelligent recognition systems with relatively simple code. It serves as an excellent learning platform for students, researchers, and software developers interested in artificial intelligence, machine learning, and image processing technologies.

Every production incident follows the same painful ritual. An alert fires at 2 am. An engineer wakes up, SSHs into a server, and begins the manual loop — pulling logs, scanning for errors, guessing what to check next. This loop can take 15 to 45 minutes before the real diagnosis even begins. Multiply that across every incident, every team, every month, and you have thousands of engineering hours lost every year to work that is repetitive, stressful, and largely automatable. I've been on that on-call rotation. I know what it costs — not just in time, but in cognitive load, in missed context, and in the compounding pressure of an active incident. So I built incopilot: a CLI tool that automates the entire first-pass triage so engineers can skip straight to actual problem-solving. This post walks through the architecture, the design decisions, and exactly how to build it yourself. Why the First 15 Minutes Are the Hardest The first phase of any incident isn't debugging — it's searching. You don't know what broke, which service is responsible, or where to look. You're making guesses based on incomplete information under time pressure. This is exactly the kind of work AI handles well: pattern recognition across large volumes of text, rapid signal extraction, and structured summarization. The key insight behind incopilot is simple: logs contain the answer almost every time. The problem is that engineers spend those first 15 minutes manually finding where the answer is buried. An automated tool can do that search in under 60 seconds. Architecture incopilot is built in Python and structured around four core modules that handle the full pipeline from log collection to report generation. Shell incopilot/ cli.py # argument parsing + console output collectors.py # journalctl, docker logs, file, bundle analyzer.py # pattern detection + line normalization reporter.py # report.md / report.json generation config.py # patterns, golden-signal map, safe-command list The collectors layer pulls logs from multiple sources — systemd journal, Docker containers, or raw log files. The analyzer applies pattern detection to identify high-signal events: OOM kills, 5xx error spikes, disk exhaustion, network timeouts, and service restarts. The reporter maps findings to the Four Golden Signals (latency, traffic, errors, saturation) and generates a structured incident brief in both markdown and JSON format. The design is deliberately read-only. incopilot never writes to your system, never restarts services, and never executes commands that modify state. This makes it safe to run during an active incident when the last thing you need is an automated tool making changes. Setup Getting started takes under two minutes: Shell git clone https://github.com/AutoShiftOps/incopilot.git cd incopilot python -m venv .venv source .venv/bin/activate pip install -r requirements.txt Quick Test Without Real Services You don't need a live environment to test incopilot. The repo includes a script that generates realistic sample logs so you can see the full pipeline working immediately: Shell python scripts/demo_generate_sample_logs.py python -m incopilot file --path sample.log ls out/ This generates a sample log file with injected failure patterns, runs the analyzer, and outputs your first incident report. The report.md file is formatted for immediate paste into an incident doc or Slack message. Running Against Real Systems For systemd journal triage — the most common production use case: Shell python -m incopilot journal --unit nginx --since "30 min ago" For Docker container triage: Shell python -m incopilot docker --container my-api --since 1h For incidents needing both sources simultaneously: Shell python -m incopilot bundle \ --unit nginx \ --container my-api \ --since-journal "30 min ago" \ --since-docker 1h What the Output Looks Like Every run produces two files in the out/ directory. report.md is formatted for immediate use — paste it directly into your incident document, Slack channel, or PagerDuty note. report.json is structured for programmatic use — POST it to a webhook, attach it to a JIRA ticket, or pipe it into a downstream automation. The report identifies detected failure patterns, maps them to the appropriate Golden Signal, lists the specific log lines that triggered each detection, and provides a plain-language summary of what the evidence suggests. Design Decisions Worth Explaining Why Python over Go or Bash? Python gives us the flexibility to integrate LLM summarization later without rewriting the tool. The current version uses pattern matching; the next version will use an LLM to generate the incident summary paragraph. Python makes that transition straightforward. Why read-only? The tool is designed to be run by any on-call engineer regardless of their familiarity with the system. A read-only tool cannot make a bad incident worse. That constraint is a feature, not a limitation. Why local execution? incopilot runs on the host or inside the container — no external API calls, no data leaving your environment. This matters for organizations with data residency requirements or security-sensitive production environments. What to Improve Next The current version is production-ready for pattern detection and report generation. The roadmap includes per-service pattern packs for nginx, postgres, Java, and Node.js applications, Slack and Teams webhook posting via a --webhook flag, unit tests with GitHub Actions CI, and an optional LLM summarization layer for human-readable incident narratives. All of this is open source at https://github.com/AutoShiftOps/incopilot. PRs and issues welcome.

Columnar storage was introduced in SQL Server 2016 as part of the SQL Server 2016 In-Memory OLTP feature. It is specifically designed for data warehousing and analytical workloads, where large amounts of data need to be scanned, aggregated, or analyzed efficiently. Columnar storage stores data in a column-wise format rather than the traditional row-wise storage, offering significant performance benefits for read-heavy operations such as reporting and analytics. Key Benefits of Columnar Storage Faster read performance: Optimized for analytics where only a few columns are needed in a query. Compression: Since column data is homogeneous, it achieves high compression rates, saving storage space. Improved query performance: Aggregating or scanning specific columns is much faster in a columnar format, especially with large datasets. Setting Up Columnar Tables in SQL Server SQL Server implements columnar storage through the Columnstore Index. The Columnstore Index is a special kind of index used in large data tables where the data is stored in columns rather than rows. The clustered columnstore index (CCI) is the preferred method when creating columnar tables. Step 1: Create a Sample Table Let's start by creating a table with a large number of rows, which we will populate with random data to demonstrate the difference between row-store and column-store formats. SQL -- Creating a traditional Rowstore Table CREATE TABLE SalesData_RowStore ( SalesOrderID INT, ProductID INT, Quantity INT, SalesAmount DECIMAL(18, 2), OrderDate DATE ); Step 2: Insert Data Into Rowstore Table For the sake of performance demonstration, we will generate a large set of random data. MS SQL -- Generate a large set of random data for Rowstore Table DECLARE @Counter INT = 0; WHILE @Counter < 1000000 BEGIN INSERT INTO SalesData_RowStore (SalesOrderID, ProductID, Quantity, SalesAmount, OrderDate) VALUES (FLOOR(RAND() * 1000) + 1, FLOOR(RAND() * 100) + 1, FLOOR(RAND() * 100) + 1, FLOOR(RAND() * 500) + 1, DATEADD(DAY, FLOOR(RAND() * 365) + 1, GETDATE())); SET @Counter = @Counter + 1; END Implementing Columnstore Index (Columnar Table) Step 1: Create a Columnstore Table Now, let's create a table with a clustered columnstore index (CCI). This index allows SQL Server to store the data in a columnar format. MS SQL -- Creating a Columnstore Table with Clustered Columnstore Index CREATE TABLE SalesData_ColumnStore ( SalesOrderID INT, ProductID INT, Quantity INT, SalesAmount DECIMAL(18, 2), OrderDate DATE ); MS SQL CREATE CLUSTERED COLUMNSTORE INDEX CCI_SalesData ON SalesData_ColumnStore; Step 2: Insert the Same Data Into the Columnstore Table You can insert the same large dataset into the columnar table in the same way. SQL -- Insert data into Columnstore Table DECLARE @Counter INT = 0; WHILE @Counter < 1000000 BEGIN INSERT INTO SalesData_ColumnStore (SalesOrderID, ProductID, Quantity, SalesAmount, OrderDate) VALUES (FLOOR(RAND() * 1000) + 1, FLOOR(RAND() * 100) + 1, FLOOR(RAND() * 100) + 1, FLOOR(RAND() * 500) + 1, DATEADD(DAY, FLOOR(RAND() * 365) + 1, GETDATE())); SET @Counter = @Counter + 1; END Query Performance Without Columnar Index Let's execute a typical query that aggregates data by ProductID and OrderDate. This will involve scanning through a large amount of data in the rowstore table. MS SQL -- Query on Rowstore Table SELECT ProductID, SUM(SalesAmount) AS TotalSales FROM SalesData_RowStore WHERE OrderDate > '2023-01-01' GROUP BY ProductID; Expected Outcome The query will scan all the rows in the table. Rowstore tables are not optimized for this type of query, and the performance might degrade with large datasets due to the need to read each row. Query Performance With Columnar Index Let's run the same query on the columnar table using a Clustered Columnstore Index. MS SQL -- Query on Columnstore Table SELECT ProductID, SUM(SalesAmount) AS TotalSales FROM SalesData_ColumnStore WHERE OrderDate > '2023-01-01' GROUP BY ProductID; Expected Outcome The columnar index stores the data by columns, and SQL Server can read only the relevant columns for the query (i.e., ProductID and SalesAmount). Columnstore indexes are highly optimized for these types of queries, resulting in much faster query execution time. Comparing the Performance of Both Scenarios To compare the performance of the two scenarios, we will execute both queries and check the execution plan and query duration. Step 1: Query Execution Plan Without Columnstore You can use the following query to view the execution plan for the rowstore table. MS SQL -- Displaying Execution Plan for Rowstore Table SET STATISTICS IO ON; SET STATISTICS TIME ON; SELECT ProductID, SUM(SalesAmount) AS TotalSales FROM SalesData_RowStore WHERE OrderDate > '2023-01-01' GROUP BY ProductID; SET STATISTICS IO OFF; SET STATISTICS TIME OFF; This will provide information on: Logical reads: The number of data pages read from diskCPU time: How much CPU time was consumedElapsed time: The total time taken to execute the query Step 2: Query Execution Plan With Columnstore Now, execute the same for the columnstore table. MS SQL -- Displaying Execution Plan for Columnstore Table SET STATISTICS IO ON; SET STATISTICS TIME ON; SELECT ProductID, SUM(SalesAmount) AS TotalSales FROM SalesData_ColumnStore WHERE OrderDate > '2023-01-01' GROUP BY ProductID; SET STATISTICS IO OFF; SET STATISTICS TIME OFF; In the execution plan for the columnstore table, SQL Server will typically show fewer logical reads and significantly lower CPU time, as it only scans the necessary columns. Performance Improvements in Columnar Tables Scenario 1: Data Compression Columnar storage achieves higher compression rates because data is stored in homogeneous chunks, which makes it more efficient in terms of storage. Compression reduces disk I/O during query execution. Scenario 2: Selective Column Scanning When querying only a few columns, columnar storage avoids scanning the entire row. In contrast, rowstore requires scanning all columns in every row, even if only a subset is required for the query. Conclusion In this example, we demonstrated how implementing columnstore indexes in SQL Server can significantly improve query performance, especially for analytics and aggregation queries on large datasets. The comparison showed that columnar storage excels in reducing query times by optimizing disk I/O, leveraging data compression, and selectively reading only the necessary columns. As a result, columnstore indexing is a great choice for data warehousing or any scenario where read performance for large datasets is critical.

By arvind toorpu CORE

In 1936, a mathematician named Alan Turing asked one of the most consequential questions in the history of computing: Can a machine think? The question has become especially popular nowadays with the development of AI. But before Turing would answer it, he had to answer a harder question: how would we know? What would count as evidence? His answer — the Turing test — was not a definition of intelligence. It was a framework for generating evidence. The question was epistemological before it was technical. It concerned not the question itself, but how we come to know an answer. Software testing is the same kind of problem. Before we can ask whether our software is reliable or safe, we must ask something more fundamental: what would it mean to know that? What counts as evidence? How do we weigh it? When is it sufficient? And when are we fooling ourselves into believing we know something we do not? These are not questions for philosophy seminars. They have direct, measurable consequences for the engineering decisions made in your organization every day. Decisions about when software ships, what risks are accepted, how incidents are explained, and where investment in quality is placed. The organizations that reason well about these questions can build systems that fail less, recover faster, and cost less to maintain over time. The ones that do not tend to confuse the absence of evidence for evidence of absence. We Learn By Testing Consider how a child learns that a surface is hot. Not by being told. Not by reading a warning. By touching it — and observing the result. The test is the touch. The evidence is the sensation. The knowledge is the updated belief about surfaces of that kind. This is not a metaphor. It is the literal structure of empirical learning. Now consider how an engineer learns that a login form works. They enter valid credentials and observe whether they are admitted. They enter invalid credentials and observe whether they are rejected. They enter a password of 10,000 characters and observe whether the system handles it gracefully. Each of these is a test. Each result is a piece of evidence. The accumulation of that evidence is what the engineer means when they say the login form "works" — not a metaphysical certainty, but a body of evidence sufficient to support the belief. Central Claim Testing is the mechanism by which we generate evidence about software behavior.Knowledge of software quality is not a property we discover — it is a conclusion we construct from evidence.That evidence is the feedback that helps us answer questions and make decisions. The quality of our evidence determines the quality of our knowledge. Poor testing does not produce uncertain knowledge. It produces confident ignorance. The last point is the one that causes the most organizational damage. Uncertain knowledge is manageable — you know what you don't know, and perhaps you can plan accordingly. Confident ignorance is dangerous because the team wrongly believes that the software is shippable. They have the test results to prove it. They ship. And then something breaks that their tests were incapable of detecting. Their tests were not designed to generate evidence about that kind of behavior. What Does A Passing Test Tell Us? One way that confident ignorance occurs is when a test passes when it should fail. A test can pass while the code it tests does not work as expected. Let’s see a few examples. The test may be making a wrong assertion — it checks that a value is returned, but not that the value is right. The test may be structured in a way that never actually exercises the code path it was written to probe. A reason for this could be a mocked dependency that silently absorbs a call that should have triggered real behavior. As a third example, a test may be correct today but pass tomorrow only because the environment has changed. It has changed in a way that masks a genuine defect: a database that was seeded with accommodating data, a clock that was frozen at a convenient moment, a network call that was stubbed before it could fail. Such tests fail to catch bugs, but more importantly, they create a false sense of quality. False evidence results in confident ignorance and, at the end of the day, unhappy customers because we’ve introduced more bugs in production. Confident ignorance also occurs when a test suite passes, and we think that we’re done. We think that we know everything. Ideally, a passing test tells us that the software did what it was supposed to do. What else is there to say? A great deal, as it turns out. Consider the following test, written for an e-commerce checkout system: Python def test_order_total_with_discount(): cart = Cart() cart.add_item(Product(name='Laptop', price=1000.00), quantity=1) cart.apply_discount(code='SAVE10', percent=10) assert cart.total() == 900.00 A passing test: the discount is applied, and the total is correct. This test passes. It passes every time you run it, in every environment, without flakiness. It is green. It is stable. And it tells you something real: that the specific combination of a £1,000 laptop, a 10% discount, and a single item quantity produces a total of £900.00. But notice what it does not tell you: What happens when two discounts are applied simultaneously.What happens when the discount code expires.What happens when the product price changes between adding to cart and checkout.What happens when the quantity is zero, or negative, or a floating-point number.What happens when the discount calculation produces a result with more than two decimal places.What happens at the boundary: does a 100% discount produce 0.00 or raise an exception? The test passed. But the space of untested behavior vastly exceeds the space of tested behavior. The passing test is not evidence that the system is correct. It is evidence that this system, with this input, at this moment, produced this output. The difference between those two things is a fundamental problem in software testing. We trust passing tests if they provide evidence. But we commit a category error when we treat the presence of passing tests as evidence of correctness in general. A passing test tells you that one path through the system behaved as expected. It may not say much about other paths. The Passing Test Trap A passing test is evidence that a specific behavior was observed under specific conditions.It is not evidence that the system generally works as expected.The gap between these two statements is the location of many production bugs. There is another reason why passing tests give us weaker evidence than we intuitively feel they do. When a test passes, it confirms something we already expected to be true. We wrote the test because we believed the system should behave that way. Confirmation of an existing belief is the weakest form of evidence, because it does not probe the limits of the belief. It merely restates it in executable form. This is why some experienced testers may feel a mild, persistent unease even in the face of thousands of passing tests. It is epistemological hygiene built through experience at work. Why Does a Failed Test Reveal More? If passing tests give us weaker evidence than we expect, failing tests give us stronger evidence than we typically acknowledge. A failing test is a learning opportunity. It tells you that you need to investigate. Does the world match your model of it, or is there something else going wrong? Failing tests in any empirical discipline are the engine of understanding. Physics advances when experiments produce results inconsistent with the prevailing theory. Medicine advances when patients respond unexpectedly to treatment. Software understanding advances when a test fails. There is a philosophical tradition — associated with the Austrian-British thinker Karl Popper — which holds that no amount of confirming evidence can prove a general statement true, but a single disconfirming observation can prove it false. A thousand white swans do not prove that all swans are white. One black swan, however, does prove that not all swans are white. The same asymmetry applies in testing. Consider this modification to the earlier example: Python def test_expired_discount_is_rejected(): cart = Cart() cart.add_item(Product(name='Laptop', price=1000.00), quantity=1) cart.apply_discount(code='EXPIRED99', percent=10) # We expect the discount to be silently ignored assert cart.total() == 1000.00 Suppose this test fails — suppose cart.total() returns 900.00. What do we now know? We know something precise and actionable: the system is applying expired discount codes. We know the specific condition under which it happens. We know the exact incorrect output. We can trace the code path. We can fix it. The failing test has given us a map to the problem. More importantly, the failing test has revealed something about the gap between our model of the system and its actual behavior. That gap is where risk lives. Every untested gap is a place where your mental model of the software could potentially diverge from what the software actually does. Failing tests can close those gaps. Passing tests, at best, could confirm that a particular gap does not exist. signalwhat it means Passing test One specific path through the system matched expectations. The model and reality agree on this point. Failing test The system's actual behavior diverged from the expected behavior. The model is wrong somewhere — and we now know where. Test that was never written No evidence either way. The gap exists, and we have chosen not to look at it. Test that was written but deleted The team once had a reason to suspect a problem here. They think that this is not the case or that the same risk is covered by another test. Flaky test The system's behavior is non-deterministic under this condition, or the test is sensitive to environmental factors it should not be sensitive to. Either is a signal worth investigating. There is a practical implication here that is often resisted: when a test fails, the correct initial response is not to fix the code. It is to understand what the failure is telling you about the system. Sometimes the code is wrong. Sometimes the test is wrong. Sometimes the requirement was ambiguous, and the failure has surfaced a genuine disagreement about what the system should do. Each of these is a different kind of finding, and treating them all as "bugs to fix" discards important distinctions. Let’s have a closer look: If a failure is caused by our code, we should go beyond just fixing the implementation. The investigation should identify why the code violated the expected behavior. Was it a faulty algorithm? An overlooked edge case? A misunderstanding of an API? A race condition? A hidden dependency? Each answer reveals something about the system’s weaknesses. It often suggests broader improvements, such as simplifying the design or adding similar tests in related areas. If the investigation stops at changing a few lines of code until the test turns green, the defect disappears, but our feedback about the system is lost. A failure caused by an incorrect test requires a very different response. The test might contain an invalid assumption or outdated expectations. It might contain unrealistic test data, an incorrect oracle, or excessive coupling to implementation details. The goal is not just to make the test green. It is to understand why the test reached the wrong conclusion, if the conclusion was actually wrong. Was the behavior intentionally changed? Did the test encode a misunderstanding of the requirement? Was it checking something that should never have been specified in the first place? Correcting the test improves not only the quality of the test suite but also the credibility of every future failure. A trustworthy test suite is built as much by removing incorrect tests as by adding new ones. Perhaps the most valuable case is when the failure exposes an ambiguous requirement. Here, neither the code nor the test is necessarily wrong. Instead, the failure reveals that different people disagree on what the system is supposed to do. Developers may have implemented one interpretation, testers another, and product owners may discover that neither matches the intended behavior. The correct action is not to modify the code or the test immediately, but to resolve the ambiguity by making the requirement explicit. Once the intended behavior is agreed upon, both the implementation and the test can be aligned with that shared understanding. In this sense, the failure has uncovered a gap in what the system does. These three outcomes illustrate why treating every failed test simply as “a bug to fix” is a missed learning opportunity. A failed test could be evidence that two views of the system disagree: the implementation, the test, and the specification can each represent a different view. Understanding which disagreement has occurred determines the appropriate action and produces knowledge that survives long after the immediate failure has been resolved. What Counts as Evidence That Software Is "Correct"? The word "correct" appears constantly in discussions of software quality. We want the software to be correct. We test to verify correctness. We define done as correct behavior. But what does correctness actually mean, and what kind of evidence is capable of establishing it? Correctness is always relative to a specification. Software is correct if it behaves in accordance with what it is specified to do. This sounds straightforward until you examine it closely, at which point three problems emerge immediately. The Specification Problem More often than not, software does not have a complete, unambiguous specification. It has a collection of requirements documents in varying states of currency. It could be user stories in a backlog or tribal knowledge in engineers' heads. It could be a legacy codebase that implicitly encodes decisions no one can fully reconstruct. The specification against which correctness would be measured is itself a moving, contested, partially-documented target. This means that when a test passes, it is agreeing to someone's interpretation of what the software should do, at the time they wrote the test, based on the information they had available. This is not nothing. But it is considerably less than "the software is correct." The Oracle Problem For a test to establish correctness, you need to know what the correct output is. This is called the test oracle — the mechanism by which you determine whether the observed output is the expected output. For simple, deterministic systems, the oracle is straightforward: you calculate the expected output and assert against it. But consider testing a machine learning model that recommends products to users. What is the correct recommendation? There is no single right answer. The oracle is probabilistic, contextual, and partially subjective. Or consider testing a distributed system under network partition conditions. The system may behave differently on different runs due to timing effects. What does "correct" mean when the system is inherently non-deterministic? The concept of correctness assumes the existence of a reliable oracle, and in many real-world systems, that oracle is unavailable, unreliable, or only partially defined. The Completeness Problem Even if you have a complete specification and a reliable oracle, you cannot test all possible inputs, states, and conditions. The input space for any non-trivial system is effectively infinite. You must select a finite subset of tests from that infinite space, and the quality of that selection determines the quality of your evidence. Consider a function that accepts a date string and returns whether it falls within a valid booking window: Python def test_date_validation(): assert is_valid_booking_date('2029-06-15') == True # valid future date assert is_valid_booking_date('2020-01-01') == False # past date assert is_valid_booking_date('2024-02-30') == False # impossible date Three tests. Passing. But the input space contains infinitely many date strings — and many interesting subsets remain untested. What about '2024-02-29' in a non-leap year? What about dates beyond the year 9999? What about the string 'yesterday'? What about an empty string? What about a date that is valid today but becomes invalid by the time the booking is processed? Each of these is a region of the input space that the three tests above say nothing about. The implication is important: correctness, as established by testing, is always partial. We can accumulate evidence for correctness in tested regions. We cannot establish correctness in untested regions. The most honest statement any team can make about their software is not "it is correct" but rather "it has behaved correctly under the conditions we have tested, and we have made a considered judgment that the untested regions pose acceptable risk." At the end of the day, it's all about confidence. What 'Correct' Actually Means in Practice Software is not correct or incorrect in an absolute sense.Software is correct relative to: (a) a known specification, (b) a specific set of tested conditions, and (c) the quality of the oracle used to evaluate those conditions.The engineering question is not 'is this software correct?' but 'what is the boundary of our evidence for correctness, and is that boundary acceptable given the risk profile of this system?' Why Do Experienced Testers Remain Skeptical? If you have spent time with genuinely skilled software testers, you have probably noticed something that can seem like pessimism or obstruction: they are never quite satisfied. A thousand passing tests does not produce for them the confidence it produces in someone less experienced. They keep asking questions. They keep looking for the untested edge. They keep saying "but what happens when..." This is not a personality trait. It is a trained posture, and it is entirely rational. The experienced tester has seen, repeatedly, the pattern by which software that appeared thoroughly tested failed in production. And it failed in ways that, in retrospect, were discoverable. They have learned, through accumulated evidence of their own, that the gap between "what we tested" and "what is possible" is always larger than it appears. Their skepticism is not a belief that the software is bad. It is a calibrated resistance to the cognitive bias that afflicts everyone who works closely with a codebase: the tendency to mistake familiarity for correctness. The Familiarity Trap You have written a piece of code and tested it many times. You’ve watched it pass in CI for months, and you develop an intuitive sense that you understand it. This sense of understanding feels like evidence of correctness. It is not. It is evidence of familiarity. The two are easily confused and devastatingly different. Familiarity narrows the space of inputs you consider. You test the paths you have already thought about. You write assertions for the outcomes you already expect. Your tests become a reflection of your mental model of the system rather than a probe of the system's actual behavior. But your mental model, however detailed, is always an approximation. This is why the most effective testers often come to a system with a degree of productive ignorance. They do not know what the system is "supposed" to do, so they are free to observe what it actually does. The Regress of Confidence There is a second reason for experienced skepticism. Adding more tests does not necessarily increase confidence. The relationship is more complex, and experienced testers have an intuitive grasp of it even if they have never formalized it. Consider a payment processing system with the following test coverage: # Test suite: 2,400 passing tests # Well-covered areas: # - Standard payment flows (card, PayPal, bank transfer) # - Refund processing # - Invoice generation # - User authentication and session management # Lightly covered areas (discovered in post-incident review): # - Concurrent payment attempts on the same order # - Payment processor timeout behavior # - Currency conversion edge cases # - State recovery after partial database write failure 2,400 tests. Green. A significant incident occurred in the lightly covered area — not in the well-covered one. The 2,400 passing tests produced genuine confidence about what they tested. But the incident did not occur in what they tested. It occurred in the gap. The experienced tester would have looked at this suite and asked: what is not here? What conditions does this suite not probe? Where is the coverage thin? The question is never "how many tests do we have?" The question is "what is the shape of our evidence, and where are the edges?" When Does Adding More Tests Stop Increasing Confidence? This is one of the most practically important questions in testing strategy, and it has no single answer — which is itself important to understand. Confidence in software quality does not increase linearly with the number of tests. It increases with the coverage of meaningful risk. These are not the same thing, and the difference between them determines whether additional testing is an investment or a cost. The Diminishing Returns of Redundant Tests Consider a login function. You write a test for a valid username and password. You write another test with a different valid username and password. And another. What confidence does each additional test produce since you are probing the same region of behavior? You are adding volume to an already-sampled area of the input space. Python # These three tests cover the same behavioral region. # Passing all three provides only marginally more confidence than passing one. def test_login_alice(): assert login('[email protected]', 'correct_password') == SUCCESS def test_login_bob(): assert login('[email protected]', 'correct_password') == SUCCESS def test_login_carol(): assert login('[email protected]', 'correct_password') == SUCCESS Redundant coverage: three tests, one behavioral region. Contrast this with the following: Python # These tests each probe a distinct behavioral region. # Each one produces a meaningful increment of confidence. def test_login_valid_credentials(): assert login('[email protected]', 'correct_password') == SUCCESS def test_login_wrong_password(): assert login('[email protected]', 'wrong_password') == FAILURE def test_login_nonexistent_user(): assert login('[email protected]', 'any_password') == FAILURE def test_login_empty_password(): assert login('[email protected]', '') == FAILURE def test_login_sql_injection_attempt(): assert login("' OR '1'='1", 'any') == FAILURE def test_login_after_account_lockout(): trigger_lockout('[email protected]') assert login('[email protected]', 'correct_password') == LOCKED Six tests, six distinct behavioral regions. Each adds genuine information. The second suite produces far more confidence. This is because confidence is a function of the diversity and relevance of evidence, not its volume. The Point of Saturation There is a point in any test suite's development where the next test you could write would probe behavior so unlikely, or so low-consequence, that the effort of writing and maintaining it exceeds the risk it mitigates. This is the point of saturation for that area of the system. Identifying that point requires judgment, not just measurement. It requires an understanding of: The risk profile of the system: what failure modes matter most, and how much?The probability distribution of actual usage: what conditions will real users encounter?The cost of a failure in each region: a failure in payment processing has a different cost than a failure in a help text tooltip.The detectability of failures in production: if a failure would be caught immediately by monitoring and easily remediated, the pre-production testing threshold can be lower. This is why testing is a risk management and confidence discipline, not a completeness and correctness discipline. The goal is not to test everything. The goal is to accumulate sufficient evidence about the right things to make an informed decision to ship — or not to ship. The Confidence Ceiling More tests increase confidence only when they probe new regions of behavior. Tests that probe already-sampled regions are not increasing confidence — they are increasing the cost of the test suite. The question 'do we have enough tests?' is unanswerable. The question 'have we generated sufficient evidence about the risks that matter most?' is answerable — and it is the right question. The Evidence Model: What Do We Actually Know? We can formulate a practical model that applies equally to a startup's first test suite and to a regulated enterprise system's quality assurance program. It applies to unit tests and to system tests, to manual exploratory testing and to AI-generated test scripts. We can call it the Evidence Model of software testing. conceptdefinition and implication Evidence region The specific set of conditions, inputs, and states that a test probes. A passing test is evidence within that region only. Evidence gap The space between what has been tested and what is possible. Every untested condition is an evidence gap. The engineering judgment is whether a gap represents acceptable residual risk. Evidence quality Determined by how precisely a test can distinguish correct from incorrect behavior. A test with a weak oracle produces weak evidence even if it passes. Evidence density The degree to which a test suite covers the high-risk regions of a system's behavior space. High-volume, low-density testing is expensive and underconfident. Low-volume, high-density testing is efficient and appropriately confident. Confident ignorance The state produced by high-volume, low-density testing. The team believes the software is well-tested. The evidence they have is real but systematically biased toward already-understood behavior. These five concepts replace the naive vocabulary of "passing tests" and "coverage percentages" with a richer language for talking about what a test suite actually knows about a system. We can use this model to evaluate specific testing decisions — what kind of evidence they generate, what gaps they leave, and whether the residual risk those gaps represent is acceptable given the context. But the model itself is the foundation. Before you can reason about testing strategy, you need to be clear about what testing is actually doing: it is generating evidence, and evidence has properties that can be evaluated. What This Means for Leadership What does this have to do with shipping velocity, team structure, or technology investment? It has everything to do with all of those things, for the following reason. Many common and costly quality failures in software organizations are not technical failures. They are situations where the team believed they knew something about their software that they did not in fact know. Testing was done, and the tests passed. The confidence was high. And then the system failed in a way that the tests were incapable of detecting. When organizations respond to these failures by demanding more tests, faster testing, or automated testing at scale, they are not addressing the cause. The questions that produce better outcomes are harder, and they require that leaders understand what testing is actually doing: Are we testing the right things, or are we testing what is easy to test?Does our test suite generate evidence about the risks that would cause the most damage if realised?When our tests pass, are we confident because we have strong evidence, or because we have a large volume of evidence in a narrow region?When our tests fail, do we learn from the failure — do we ask what it reveals about the gap between our model and reality — or do we simply fix it and move on?When we add AI-generated tests to our suite, what evidence are those tests generating, and what regions of risk do they leave unprobed? The answers to these questions require clarity. And clarity is necessary in a world where AI is reshaping how code is written and what testing looks like. However, it cannot reshape what testing is for. Wrapping Up In software, a great deal of what we know comes from testing. The CTO knows whether the platform can absorb next quarter's growth because load tests and failure simulations produced evidence about its limits. The product owner knows whether a feature is ready for customers because acceptance criteria were checked against observed behavior, not assumed behavior. The developer knows whether a refactor preserved existing functionality because a regression suite ran and reported back its feedback. The tester knows where the system is fragile because exploratory sessions and structured test design surfaced behavior no one had anticipated. The DevOps engineer knows whether a deployment is safe to proceed because monitoring, canary analysis, and synthetic checks are themselves a form of testing in production. They generate evidence in real time about how the system behaves under real conditions. Each of these roles is asking a question, at a different altitude, with different consequences attached to the answer. But the underlying mechanism is the same at every level of the organization: testing produces evidence, evidence becomes feedback, and feedback is what we use to decide what to do next. The CTO's decision to invest in a new architecture. The product owner's decision to ship or hold. The developer's decision to merge. The tester's decision to escalate a risk. The DevOps engineer's decision to roll back. None of these are made in a vacuum. They are made on the strength of the evidence available at the moment the decision is required. Testing, understood this broadly, is not a phase or a department. It is the organization's backbone for converting uncertainty into decisions it can support confidently.

By Stelios Manioudakis CORE

Most of what AI promises in security rides on something duller than the model itself: whether it can see the environment it's defending. When it can't, a stronger model doesn't help. It makes the gaps harder to spot, and it brings a few new ones of its own. Two figures from the past year carry most of the story. CrowdStrike's 2026 Global Threat Report put the average eCrime breakout time at 29 minutes for 2025. (Breakout time is the stretch between an attacker landing on one host and pivoting to a second.) That's down from 48 minutes the year before, 62 in 2023, and 98 in 2021. The fastest single case clocked 27 seconds, and in one intrusion, data was already moving out four minutes after the attacker got in. The report also found that 82% of intrusions involved no malware at all, with the attacker simply logging in on valid credentials, and that activity from AI-enabled adversaries climbed 89% over the prior year. Microsoft's 2024 Digital Defense Report says where most of those footholds come from. Among ransomware attacks that reached a ransom demand, north of 90% rode in on an unmanaged device, used either for the initial access or for the encryption itself. The typical breach starts on hardware that the security team has no eyes on. Stack those together, and the shape is hard to miss. Attackers are inside and moving within half an hour, and they come in through whatever the inventory missed. So one unglamorous question matters more than any of the ones about detection speed: can the AI see what it's supposed to be guarding? If the answer is no, a more capable model won't save the situation. It mostly produces a more polished account of the same gap. Loud Failures Are the Easy Ones The failures that make noise are the ones we handle well. A bad model output is obviously bad, and an engineer overrides it. An integration breaks and throws an error. A pipeline job dies. A dashboard goes red, and somebody gets paged. All of it is visible, and visible problems get worked on. Missing telemetry is the other kind of failure. It's quiet. Feed an AI system partial data, and it still hands back a clean summary of your risk: it scores assets, ranks incidents, and tells you the controls look fine, working only from the systems that happen to report in. The write-up reads well, the confidence figure looks earned, and nothing in it mentions that a third of the fleet was never in the picture. This cuts deeper with AI than with the dashboards we're used to, because of how the output gets read. A conventional dashboard makes you look at the raw material: the filters, the timestamps, the source systems, the columns that came back empty. The gaps are at least in front of you. An AI summary compresses all of that into a paragraph. When the data underneath is complete, that's a gift to a tired analyst. When it isn't, the compression is what buries the gap. And a confident wrong answer is harder to deal with than an honest "I'm not sure," because uncertainty at least sends someone to go look. A green light sends them home. The “All Good” Trap Picture a concrete version. An AI dashboard reports the internet-facing services as healthy and low risk. A senior engineer who knows the estate reads that as handled. Then someone pulls the asset count and finds 71% of that class is actually instrumented. The 71% was scored correctly; the trouble is the other 29%, which the model had no reason to mention and which everyone has now stopped worrying about, because the tile was green. That missing slice is hardly ever random. It collects the awkward stuff: contractor-managed endpoints, cloud resources nobody claimed, an old VPN path, a few personal laptops, service accounts that outlived their owners, an integration wired up before the current identity platform existed. None of it reaches the model unless something feeds it telemetry, and the model has none of the institutional memory a long-tenured engineer carries: which subnet belongs to a vendor, which credentials should have died two reorgs ago. It works with what it was given. If that input is partial, the output still looks whole, and that mismatch is where the danger sits. The Control That Was Never Switched On The 2024 Change Healthcare breach is the clearest case I know of for why "we have that control" and "that control is doing anything" are separate claims. It became the largest healthcare data breach in U.S. history, hitting roughly 190 million people, nearly one in three Americans. How they got in was unremarkable. Attackers used stolen credentials on a Citrix remote-access portal with no multi-factor authentication turned on, spent about nine days moving around inside, pulled out terabytes of data, and only then dropped the ransomware. The part worth sitting with came out in the parent company's testimony to Congress: MFA was company policy across external-facing systems. The control was real on paper. It just wasn't switched on for that one portal, and nobody noticed the hole until it got used. Run an AI risk model over that environment a week earlier. It finds a documented MFA policy and marks identity coverage as good. Unless it was built to check whether MFA is actually enforced on each external endpoint, rather than whether a policy exists somewhere, it shows green. The control was present and useless at once, and a model reading policy documents instead of a live enforcement state would have signed off. No alert is not the same as no problem. Often, it just means nothing was watching that spot. The HIPAA Rewrite Asks for Visibility Before Anything Clever Change Healthcare didn't only cost money; it moved policy. In January 2025, the U.S. Department of Health and Human Services proposed reworking the HIPAA Security Rule for the first time in more than twenty years. The comment window drew close to 5,000 responses, and a final version is expected around 2026. The telling part is the order. Before anything sophisticated, the draft would require a current, yearly-updated inventory of every technology asset that touches protected health data, plus a network map; MFA with only narrow exceptions; encryption in transit and at rest; vulnerability scans twice a year; an annual penetration test; and network segmentation. It would also scrap the old split between "required" and "addressable" safeguards and make nearly everything mandatory. That last move is the regulatory echo of the point above: "addressable" is exactly how a safeguard ends up written down but never enforced, which is the road the Citrix portal took. The reaction confirmed what practitioners keep saying about money. HHS estimated a first-year cost near $9 billion, and an industry group led by CHIME, joined by more than a hundred hospital systems, asked the administration to pull the rule, arguing that smaller and rural providers simply cannot carry it. Many healthcare organizations spend something like 80% of their budget on infrastructure and only a sliver on security. The proposed rule tells them the sliver has to go first to seeing the estate and enforcing the basics, not to one more detection layer bolted on at the end. Make the Score Show Its Coverage So what do you do about the silent-failure problem? Not stop using AI summaries. Make every summary carry its own coverage, so the gap rides along with the number instead of getting dropped on the way into a slide. Most risk summaries today hand back something like this: { "asset_class": "internet-facing-services", "risk_score": 18, "status": "healthy" It's tidy, and it tells you almost nothing, because you can't see whether it rests on full data, partial data, week-old data, or just the systems that were easy to wire up. A coverage-aware version states its own basis: { "asset_class": "internet-facing-services", "risk_score": 18, "status": "healthy", "coverage": { "assets_known": 412, "assets_instrumented": 293, "coverage_pct": 71, "uninstrumented_pct": 29 }, "data_freshness": { "newest_signal": "2026-06-21T09:14:00Z", "oldest_signal": "2026-06-09T22:40:00Z", "stale_sources": ["byod-mdm", "contractor-vpn"] }, "confidence_basis": "computed on 71% of known assets; excludes BYOD and contractor access", "blind_spots": ["unmanaged-endpoints", "third-party-saas"] } Now status: healthy next to coverage_pct: 71 reads completely differently, and you don't have to dig to get there. Nobody needs perfect coverage; the last few percentage points cost a fortune to instrument, and plenty of assets don't justify it. What you want is a system honest about where it's blind: what it covered, what it skipped, how old the freshest gap is, and what the score rests on. Any AI security view headed for a decision-maker should answer those four on its own. A confidence number that can't say where it came from is decoration. Shadow AI: The Gap You Didn’t Know You Opened Up to here, the unseen assets have been familiar ones: endpoints, accounts, third-party links. The fastest-growing blind spot in 2026 is newer, and most security teams opened it without meaning to. It's the AI tooling their own people already use every day. The numbers are blunt. IBM's 2025 Cost of a Data Breach report found one in five breached organizations was hit through shadow AI, meaning unsanctioned generative-AI tools nobody in security signed off on, and that shadow AI added roughly $670,000 to the average breach. Netskope counted the distinct generative-AI apps in use across enterprises rising past 1,550 over the year, from around 317 at its start, with close to half of users reaching them through personal accounts nobody is watching. One survey put the share of organizations with no real view of how data moves in and out of AI tools at 86%. IBM filled in the reason: 63% have no policy for managing AI or heading off shadow use, and among firms that took an AI-related hit, 97% had no proper access controls on AI in place. It's the same visibility problem in different clothes. An engineer pastes proprietary code into a chatbot to debug it; a manager drops a customer list into an unapproved summarizer. The data crosses the boundary and settles into a third-party model the security team can't see into, govern, or claw back. A risk model has nothing to score, and the data path shows up in no inventory. Banning the tools doesn't help much; people keep using AI after a ban, which only drives it further out of sight. The workable answer looks like the old shadow-SaaS hunt: discover what's actually in use, give people sanctioned options so they don't need the back channels, and put data-loss controls on the paths out. IBM's analysts put it bluntly that the human-centered measures, the training sessions, warning emails, and written policies, fail over and over, and the only thing that reliably holds is a technical control that stops the upload before it leaves. The Defender’s AI Is Part of the Attack Surface Now There's a second new gap, and it's the AI you brought in to help. The moment a model is wired into your environment, it becomes one more thing to watch and fence in, and it drags a crowd of machine identities along with it. Non-human identities, the service accounts and API keys and OAuth tokens and workload credentials, and now the AI agents, already outnumber human users badly: estimates run from about 45 to 1 in an ordinary enterprise to 144 to 1 in cloud-native and DevOps shops, and one vendor's count grew 44% in a single year. A typical enterprise has gone from tens of thousands of machine identities a few years back to something like a quarter million today. CyberArk's 2025 identity survey found 68% of organizations with no identity-security controls for AI and 47% unable to lock down shadow AI at all, while SpyCloud researchers turned up 6.2 million exposed credentials tied to AI tools in one year. Most are created outside any IT process, hold broad permissions, and never get revoked. Together, they're the least-governed part of the modern attack surface, and because machine-to-machine traffic looks like business as usual, abuse of them goes unseen until the data is already gone. Agents push this up another level. Gartner expects task-specific AI agents in 40% of enterprise applications by the end of 2026, against under 5% in 2025. A static API key has a fixed scope you can list and audit. An autonomous agent calls external APIs, spins up sub-agents, writes and runs its own code, and can pick up new permissions while it runs, so you can't fully say ahead of time what it will touch. And the model in the middle of all this carries a weakness you can't patch away. Prompt injection has held the top slot in OWASP's Top 10 for LLM Applications across two editions, for a structural reason: a model takes instructions and data over the same channel and can't reliably tell which is which. Tuck an instruction inside a document, a support ticket, a web page, or a code comment, and a model that reads it may follow it as a command. Retrieval-augmented generation doesn't close that, and neither does fine-tuning; OWASP's advice is layered defense, with least-privilege tooling, filtering in and out, a human in the loop for anything sensitive, and regular adversarial testing. This isn't a thought experiment: CrowdStrike worked incidents at more than 90 organizations where attackers went straight at AI tools and dev platforms, slipping malicious prompts into live systems, and mentions of ChatGPT in criminal forums jumped more than fivefold. Which brings it back around: if you can't watch and constrain the security AI itself, it stops being a defense and turns into another asset on the surface, one that can be talked into helping the other side. There’s a Name for the Attacker’s-Eye View: CTEM That better question, asking what you look like to an attacker right now instead of whether you're safe, already has a formal home. Gartner called it Continuous Threat Exposure Management, or CTEM, in 2022, and has since put it among its top security investments for 2026. CTEM is a program, not a tool you buy. It cycles through five stages, scoping, discovery, prioritization, validation, and mobilization, and the heart of it is reasoning about attack paths rather than counting isolated vulnerabilities. Scoping opens right where this article does, by deciding which slices of the attack surface actually matter to the business, since not every system carries the same weight. Discovery is where continuous asset inventory lives, and the tooling is worth knowing by name: Cyber Asset Attack Surface Management (CAASM) pulls asset data out of the systems you already run and stitches it into one view, while External Attack Surface Management (EASM) shows what's reachable from outside. Validation is where you test whether a control actually fires, the discipline that catches a documented-but-disabled MFA setting. Gartner has predicted that organizations running an exposure-management program would be three times less likely to be breached, though that's a forecast, not a measured result; the independent reads so far show better visibility for adopters, not a proven drop in breach rate. The direction holds up regardless, and the practical value is that it turns "what do we look like to an attacker" from a one-time exercise into a standing process. Push the Checks Earlier, Not Later None of this holds if security stays a last-step review. AI is speeding up how fast teams turn out code, tests, and operational glue, and that same speed carries mistakes and untested assumptions downstream just as quickly. Adding observability at the very end only ships the risk faster. The fix is to pull the security and observability questions into the definition of done, so a feature isn't finished until it can answer them. How will this get monitored in production? What does it emit, who can reach it, and what data does it handle? What does abuse look like in the logs, what happens when something upstream falls over, and how does the system flag that it doesn't have enough context to judge? AI helps with a lot of this, drafting tests, spotting risky dependencies, reading code paths, but AI-assisted development with no AI-assisted assurance just moves risk along faster. The checking has to keep pace with the generating. And none of it stands in for zero trust; it makes zero trust matter more. Don't extend trust on the basis of network, device, or past access. Verify as you go, hold access down, watch how things behave, keep the blast radius small. A model can narrate your risk in fluent prose and still do nothing to reduce it without real identity, policy, and enforcement underneath. What the Breach Math Says If the case needs a number, IBM's 2025 report supplies one with an edge on both sides. The global average breach cost dropped 9% to $4.44 million, the first decline in five years, on faster detection and containment from AI-assisted defense. Organizations using security AI and automation heavily spent about $1.9 million less per breach and cut roughly 80 days off the breach lifecycle. So AI, on a foundation that works, does pay. The same report shows the other path. Shadow AI added that $670,000; breaches involving it ran longer and gave up more personal data and IP; and again, 97% of organizations hit by an AI-related incident had no basic access controls on AI. Only about 17% have automated blocking and scanning for AI use, which leaves everyone else on the human-centered controls that don't hold. In the U.S., the average breach reached a record $10.22 million, driven up by regulatory and escalation costs. The through-line matches everything above: AI defense built on real visibility and governance saves real money, while AI rushed in over a blind spot costs more than skipping it would have. Whatever foundation it lands on, the technology makes it bigger. What to do on Monday If you're running an engineering or security team, five moves beat buying another tool. Make one AI security view honest about its coverage. Take a single AI-generated dashboard and have it show coverage and freshness next to every score, so it can't call something healthy without saying what it actually looked at. Check enforcement, not policy. Pick one control that matters, MFA on external access being the obvious one, and line up what the policy says against where it's truly enforced today. The Change Healthcare gap lived in exactly that distance. Count your shadow AI and your machine identities. Find the unsanctioned AI tools in play and the data routes into them, and inventory the service accounts, tokens, and agents running in your environment. Nothing you haven't counted can be governed. Drill against the clock. With breakout time around half an hour, rehearse a believable initial-access path and ask what the attacker reaches in the first thirty minutes, and which telemetry would confirm or rule out each step. Make it recurring, not a one-off. Run a visibility-focused Wheel of Misfortune. Pick a plausible gap, an unmanaged endpoint, a stale cloud asset, an agent fed untrusted content, and ask whether your AI security view would even see it, whether it would call out the missing data, or whether it would just hand back a confident summary anyway. You're hunting for what the dashboard can't see, ideally before someone else finds it first. The First Question AI is going to be a permanent fixture in security. It will cut manual work, add context to noisy signals, and let teams respond with more than instinct, and the cost data shows that the payoff is real. What it won't do is let you off the hook for the fundamentals. It raises the price of skipping them, partly because it makes a thin foundation look finished, and partly because the AI itself becomes one more thing you have to see and hold in check. If you can't see your assets, your identities, your telemetry, your shadow AI, and the gaps in your controls, AI won't cover that distance. It mostly paints over it. The first question for AI security was never how powerful the model is. It's whether you can see enough of your own environment to put it to work.

If you have ever tried to build a “mortgage rates today” feature, you have probably landed on the same fork in the road I did. Consumer finance sites publish attractive numbers, but those pages are editorial HTML, not APIs. Layout changes break scrapers, terms of use are unclear, and it is hard to explain what statistic you are actually showing. For a developer project meant to be forked on GitHub, that path is a dead end. I wanted something smaller and more honest: national 30-year and 15-year fixed benchmarks from a source economists already cite, plus a separate surface where first-time buyers could ask basic questions without handing over sensitive data. The result is US Homes Mortgage Agent, an open-source FastAPI application with two clear jobs. One page shows macro rates and a chart. Another page hosts a conversational assistant that is grounded in the same data and in plain Python math. This article explains the problem framing, the architecture, and the implementation choices that kept the project maintainable. The Problem: Rates Pages Are Not Data Products Headline mortgage rates on news sites are useful for humans scanning a story. They are awkward for software. The number in the article may come from a weekly survey, a lender panel, or a marketing table, and the HTML around it changes often. Automating that extraction couples your app to someone else’s front-end design. There is a second trap that is easy to miss. The most common free national series for conventional fixed rates, Freddie Mac’s Primary Mortgage Market Survey, is published weekly through the Federal Reserve Economic Data service (FRED). Series MORTGAGE30US and MORTGAGE15US are the ones you will see referenced in macro commentary. If your UI promises “today’s rate” and refreshes every morning, users will still see the same value for several days. That is correct behavior given the underlying data, but it looks like a bug unless you show the observation date. First-time buyers add a different problem. A chart alone does not answer “what is pre-approval?” or “what is the difference between principal and interest and a full housing payment?” A chat interface helps only if you constrain it. Letting a model invent rates or amortization is worse than no chat at all. What the Application Does The app splits responsibilities instead of blending them into one screen. On the rates and chart view, the server exposes the latest national benchmarks and a history chart with ranges of 7 days, 30 days, 90 days, and one year. Data comes from FRED. A scheduled job runs every day at 9:00 in a configurable timezone (I use America/New_York by default) and stores a snapshot in SQLite. That snapshot powers the headline tiles and feeds one of the assistant’s tools. From the first-time buyer's view, the user gets a simple chat UI. Messages stay in the browser for the session. The server does not write chat history to the database. When the user sends a message, the backend calls an LLM with a system prompt oriented toward education, not legal or tax advice, and with two callable tools: one to read the current benchmark snapshot, and one to compute fixed-rate principal and interest with standard amortization formulas. The product is intentionally not a lender marketplace, not a credit product, and not personalized underwriting advice. It is a reference implementation for macro transparency plus responsible LLM integration. Why FRED, and Why SQLite Only for Benchmarks FRED gives documented time series with a free API key. For this project, that was the right trade: legal programmatic access, stable identifiers, and enough history for charts without building an ETL pipeline on day one. The SQLite database holds one concern: daily benchmark snapshots. Each successful refresh upserts a row keyed by calendar date with 30-year and 15-year values, the FRED observation dates for each series, and a fetched_at timestamp. Chart data for longer windows is fetched from FRED on demand when the user changes the range. I accepted extra API calls on chart loads in exchange for not maintaining a large local history store. Chat content never lands in that schema. If you are cloning the repo, your local data/ directory is git ignored along with .env, so you start fresh without accidentally publishing keys or a test database. Architecture at a Glance The deployment model is deliberately boring: one FastAPI process, static assets, Jinja templates, and an in-process APScheduler cron job. That keeps the project easy to run locally with uvicorn and easy to reason about for readers who want to fork it. Logical component diagram: browser, FastAPI, SQLite, FRED API, and LLM HTTP API At runtime, three paths matter. Page load for rates. The browser requests HTML, then calls /api/rates/today for the cached snapshot and /api/rates/chart for series data. If FRED is slow on a 90-day or one-year range, the UI shows a loading state and disables the range control so users know the chart is working. Scheduled refresh. At 9:00 local time, the job pulls the latest observations for both series and upserts the snapshot. Between weekly FRED releases, the values may not change. The UI still benefits from a daily pull because you pick up new observations as soon as they exist. Assistant chat. The client posts the in-memory conversation to /api/chat. The server prepends a system prompt that includes the current benchmark context, calls the LLM with tool definitions, and loops if the model requests tool execution. When tools return JSON, the model produces the final natural-language answer. Grounding the LLM With Tools The pattern I cared about most was keeping numbers out of the model’s imagination. The assistant exposes two tools. get_benchmark_rates reads the same SQLite snapshot as the dashboard tiles and returns structured JSON with rates and observation dates. estimate_monthly_pi takes the loan amount, annual rate, and term, then runs deterministic amortization in Python. Taxes, insurance, and PMI are out of scope for that calculation, and the prompt tells the model to say so. Here is the core amortization helper. It is small on purpose: easy to test, easy to audit. def monthly_principal_interest(loan_amount, annual_rate_percent, years): n = years * 12 r = (annual_rate_percent / 100.0) / 12.0 if r == 0: return round(loan_amount / n, 2) payment = loan_amount * (r * (1 + r) ** n) / ((1 + r) ** n - 1) return round(payment, 2) The system prompt reinforces guardrails: stay educational, refuse to handle sensitive identifiers like SSNs, and point users toward HUD counselors or licensed lenders for personal decisions. That does not replace compliance review if you productize this, but it sets a baseline for an open demo. LLM Provider Choice The reference code uses OpenAI’s Chat Completions API with OPENAI_API_KEY and defaults the model to gpt-4o-mini. The HTTP client posts to OPENAI_BASE_URL, which defaults to https://api.openai.com/v1. In practice, any host that implements a compatible chat completions endpoint and supports tool calling can be configured the same way, including several OpenAI-compatible gateways. You should verify tool-call behavior and error formats on your chosen provider before relying on it in production. The rates dashboard works with only FRED_API_KEY. The assistant returns a clear configuration error until an LLM key is present, which makes local development predictable. Operations and Open Source Hygiene For a public GitHub repository, secrets stay in .env. FRED and LLM keys are required at runtime but must not be committed. SQLite files live under data/ and are ignored by git. If you deploy behind a reverse proxy, terminate TLS there and probe /api/health. One caveat for horizontal scaling: APScheduler runs inside each process. Multiple replicas without coordination will each run the 9:00 job. For a demo, that is harmless. For production, you would externalize scheduling or elect a single worker for the cron task. LLM calls have cost and retention implications on the provider side. Even though this app does not store chats server-side, the provider still processes prompt content under its own policies. Plan for that if you expose the assistant publicly. What I Would Extend Next National weekly averages are the right free starting point, not the final word. State-level pricing, daily lender indices, and streaming chat responses are all reasonable extensions, each with its own licensing or infrastructure cost. Automated tests against mocked FRED responses and golden tests for the amortization helper would be the first engineering upgrades I would prioritize. Closing Thoughts The useful analysis from this build is the separation of concerns. Macro data comes from FRED with explicit observation dates. Education flows through an LLM that is allowed to explain and summarize, not to invent rates. Math runs in code the way it always should have. If you want to explore the implementation, the project is open source. Clone it, add your own API keys, and treat it as a starting point rather than a finished financial product.

The adoption of retrieval-augmented generation (RAG) from research papers to production systems has been rapid. Those who tried it in 2023 are now deploying it at scale for enterprise search, internal knowledge bases, and customer-facing assistants. However, a lot is still between a working prototype RAG and one that can withstand traffic on the road, using real data, and real modes of failure. This article explains what this gap is, how to plug it, and where most production pipelines fail. What a Production RAG Pipeline Involves A basic RAG pipeline consists of three components: a document store, a vector database to store embeddings, and a language model to produce answers based on the retrieved context. There, most of the tutorials end. Production systems have no such convenience. A production pipeline also needs: Ingestion pipelines that process document updates, deletions, and format variationsSemantic preserving chunking strategies for different document types.Incorporating models that align with the retrieval use case (asymmetric vs. symmetric search).A retrieval layer to support hybrid search, metadata filtering, and re-rankingA generation layer that has prompt management, output validation, and fallback behaviorEvaluation is hooked at each stage so that one can catch the degradation in time for the users It is easy to make each piece individually. The engineering challenge is to get them to cooperate under production conditions. Choosing the Right Vector Database Everything downstream is influenced by the choice of the vector DB. The top choices for 2026 are Pinecone, Weaviate, Qdrant, Milvus, and (for teams already using Postgres) pgvector. A few things that matter more than raw benchmark performance: Filtering support. If your documents have metadata (date, category, author, department), then you require a vector DB that filters pre-retrieval and not post-retrieval. Recall precision is destroyed by post-retrieval filtering. Both Qdrant and Weaviate do a good job of this. In recent versions, Pinecone introduced support for metadata filters, though there are edge cases for large sets of filters. Update semantics. Some vector DBs are delete-and-insert, which means there may be retrieval gaps while they are reindexing. In some cases, such as summarizing news or for internal documentation, this behavior can result in stale or missing results. Compared to most, Milvus performs rolling updates more gracefully. Hybrid search. Dense retrieval would not find exact matches for keywords. Sparse retrieval (BM25-style) ignores semantic similarity. Most production teams ultimately choose to use both, employing a fusion layer (typically the default Reciprocal Rank Fusion). Weaviate has this integrated. For others, it is wired up by yourself. Scale and cost. pgvector performs well up to ~1-2 million vectors on reasonable hardware. In addition, a specialized vector database pays for itself. Chunking Strategy: The Part Most Teams Get Wrong The way you partition documents to be embedded affects the quality of retrieval more than any other choice. The simplest method is fixed-size chunking with a token limit and yields fairly poor results for a structured document such as a PDF, a contract, or technical specifications. Better approaches: Recursive character splitting with overlap preserves context across chunk boundaries. LangChain's RecursiveCharacterTextSplitter is a typical first step, with chunk sizes of 512-1024 tokens and an overlap of 10-15%. Semantic chunking is similar to grouping sentences by embedding similarity (not token count). This is more suitable for conversational transcripts and prose-heavy documents. The downside is that it takes longer to calculate when ingesting. Document-aware chunking parses the structure before splitting. If HTML or Markdown, you split on headers. In the case of PDFs, you use layout analysis to separate tables, figures, and body text. Libraries like unstructured.io handle much of this automatically. A pattern that should be adopted early: Keep the entire document with the chunk. At the time of retrieval, you retrieve the chunk to score for relevance, and if needed, you pull some of the context from the parent document to generate. This is known as a parent-child retrieval pattern, and is a great way to get higher quality answers to longer context queries without increasing the size of your index. Re-Ranking: The Layer Most Prototypes Skip Vector similarity search returns the top-k most similar chunks. That's not necessarily the top-k most useful chunks that answer a specific query. Instead of scoring each individual piece of information, cross-encoder re-rankers process the query and the information to be retrieved as pairs and return a relevance score that considers the relationship. Models like cross-encoder/ms-marco-MiniLM-L-6-v2 are small enough to run in real-time and measurably improve precision on mixed-topic corpora. Agentic RAG setups take this one step further in 2026. The agent analyzes the results retrieved, and if they are not enough, the agent will perform the retrieval again. This iterative pattern is for multi-hop questions that can't be answered by a single retrieval round. For enterprise knowledge applications with a focus on the quality of the answer, rather than latency, teams developing with LangGraph or CrewAI are turning to this pattern. Prompt Management and Output Validation The generation layer fails in expected ways: it hallucinates when the retrieved context is irrelevant, it hedges when there is conflicting retrieved context, or it simply ignores the context and generates from training data. There is no universal solution to each failure mode. Add a faithfulness check to the hallucination. Once the generation is complete, have the LLM perform a separate call to check if the answer is based on the retrieved passages or not. This is a metric that is provided by tools such as RAGAS and can be tracked over time. Don't hardcode prompts in application code for prompt management. Having the ability to perform rapid iterations of a prompt without redeployment, and identify which iteration was active at the time of an incident (a prompt registry, even a basic one in a config file or database) is a huge plus. Evaluation: The Missing Foundation The majority of RAG pipelines are delivered without any systematic evaluation. The teams that end up in trouble 6 months later are the ones that have neglected to do this. The minimum viable evaluation setup includes: Golden Data Set: 100-200 representative question-answer pairs from real use.Metrics for retrieval (recall@k, MRR) and generation (faithfulness, answer relevance, context precision)A nightly or weekly eval run that compares current performance against a baseline Both RAGAS and TruLens connect to popular RAG frameworks and provide you with the above metrics without having to build them yourself. Where Teams Often Need Outside Help When creating RAG pipelines for the first time, teams often overlook the operational aspects, such as implementing model versioning, handling migrations of indexes when switching models, and tracking retrieval drift as document corpora expand. As companies transition to production-level generative AI development, the combination of in-house engineering and generative AI consulting can rapidly address these gaps before hiring and training all the necessary personnel. It is where generative AI integration services can contribute the most value: from architecture to production, running and monitored systems. When getting started on the selection process for a vector DB, the chunking strategy, and evaluation infrastructure, ensure that you have the correct setup for a production RAG system. Those decisions will be much easier to get right from the start rather than after your index is up and running.