AWS Certified Data Engineer - Associate

You are working as a Data Engineer at a healthcare company that captures real-time patient data from numerous medical devices. The data from these devices are streamed continuously to an Amazon Kinesis Data Stream. To ensure low latency and durability, you decide to distribute the data to multiple downstream services for immediate processing. Several machine learning models and storage solutions need to consume this data simultaneously with different processing workloads. Due to the varying requirements, you aim to achieve a fan-out pattern. Furthermore, alerts and notifications need to be sent to healthcare professionals based on specific thresholds. You decide to use Amazon SNS for this purpose. What is the most appropriate way to set up the system to achieve this?

As an AWS Certified Data Engineer, you have been tasked with creating a data API to make customer data available to other systems in real-time. Your company uses Amazon RDS for PostgreSQL to store customer data. You need to ensure that the data transformation and processing are efficient and that other systems can consume the data with minimal latency. How can you best achieve this within the AWS ecosystem?

You are a Data Engineer at a company that processes large volumes of data. As part of your data pipeline, you have a process that ingests data from various sources and transforms it before storing it into your data warehouse. You are using Amazon SQS to manage the ingestion of data by queuing messages that represent the data files to be processed. Due to the need for real-time processing, you must ensure that if any part of the system fails or encounters an issue, the appropriate team is immediately alerted so they can address the problem. Which service would you use to send these notifications, and how would you integrate it with Amazon SQS?

You are a data engineer working for a media company that processes a large volume of user-uploaded video files. Your task is to create an AWS Lambda function that processes these video files as soon as they are uploaded to an Amazon S3 bucket. You need to ensure that the Lambda function has access to a temporary storage volume for buffering data during processing. Which of the following steps should you take to achieve this?

You are a data engineer for a financial analytics firm, responsible for managing and analyzing large datasets. The firm’s transactional data is stored in an Amazon Aurora MySQL database. The analytical workload has increased significantly, and you need to speed up your data queries. You decide to store a subset of this data in Amazon Redshift for faster querying and reporting. What is the most efficient method to move and keep this data in sync while ensuring minimal impact on the Amazon Aurora database performance?

You are an AWS Certified Data Engineer working for a retail company. Your task is to catalog data stored in Amazon S3 so that it can be queried and analyzed easily. You choose to use AWS Glue for this task. You want to create a new connection in AWS Glue to your Amazon S3 bucket named 'retail-sales-data'. What are the necessary steps you should follow to successfully create this connection and catalog the data? A. Create an AWS Glue Data Catalog, then configure a Crawler to scan the Amazon S3 bucket and store the metadata in the Data Catalog. B. Upload the data to Amazon S3, create an Amazon RDS database, and configure the AWS Glue Crawler to use the RDS instance. C. Use AWS Glue’s ETL (Extract, Transform, Load) feature to transform the data in Amazon S3 and move it to Amazon Redshift. D. Download the AWS Glue data plugin and configure it to connect to the Amazon S3 bucket directly from your local machine.

Alex, a Data Engineer at a financial services company, has been tasked with optimizing the storage cost management of their AWS environment. The company uses Amazon S3 for storing large volumes of transaction logs and DynamoDB for storing metadata about these transactions. In order to comply with regulatory requirements, the transaction logs must be kept for a minimum period, but after that, they can be moved to cheaper storage or deleted. Alex implements S3 versioning to protect against accidental deletions or overwrites and uses DynamoDB Time to Live (TTL) to automatically delete entries after their retention period expires. To further optimize storage costs, Alex wants to automate the process of moving older S3 objects to cheaper storage classes and eventually delete them. What should Alex do to optimize the lifecycle of the data in S3?

As a data engineer, you've been tasked with setting up a data pipeline to process and transform large sets of data from various sources. The data needs to be analyzed and used to train machine learning models in Amazon SageMaker. To design your data models and manage schema evolution effectively, you must establish data lineage throughout the pipeline to ensure data quality and traceability. Which AWS service should you use to automate the extraction, transformation, and loading (ETL) of data while also providing built-in capabilities for tracking data lineage?

You are a data engineer at a company that collects large volumes of data from IoT devices. The data is streamed into an S3 bucket in near real-time. Your task is to automate the processing of this data using AWS services. The processing needs to be triggered at the arrival of new files in the S3 bucket, and you have decided to use AWS Batch for the processing jobs. Which combination of AWS services can be used to automate this data processing workflow?

You are a data engineer at a retail company, and you are conducting an analysis of customer purchase data stored in Amazon S3. You decide to use AWS Glue to clean and catalogue the data and then explore it using Amazon Athena notebooks powered by Apache Spark. You want to aggregate customer purchase amounts per region and create visual insights on customer spending trends. Which of the following steps should you follow to achieve your goal efficiently?

You are a data engineer at a company that processes massive amounts of log data generated by various microservices. The log data is stored in Amazon OpenSearch Service for real-time analysis and monitoring. Recently, you have observed that the OpenSearch cluster is running slower because it is overwhelmed with the growing number of logs. You need a solution to maintain and monitor your data pipeline effectively ensuring better performance of the OpenSearch cluster. What should you do?

During a routine audit in your data processing pipeline, you discovered inconsistencies in the data stored in your Amazon S3 buckets. You suspect that the data transformations performed by AWS Glue DataBrew might be the root cause of the inconsistencies. As a Data Engineer, you need to investigate and ensure data quality in your pipeline. Which of the following approaches would be the most effective to identify and rectify the data inconsistencies?

You are a data engineer at a financial services company and are responsible for setting up a secure data pipeline. You've been asked to grant access to a specific Amazon S3 bucket only to your analytics team so they can run queries for the quarterly financial reports. The team members should only have read access to the bucket. You already have an IAM group named 'AnalyticsTeam'. To ensure security, you decide to use AWS PrivateLink to keep the data transfer within the AWS network. What is the best way to achieve this?

You are a data engineer at a company that uses Amazon Redshift, Amazon EMR, and Amazon S3 for big data analytics. To enhance your data security and governance, you are considering using AWS Lake Formation for managing data access permissions. A specific requirement is to ensure that the data engineers can run jobs on Amazon EMR clusters and have restricted access to only specific datasets stored in Amazon S3. How should you configure Lake Formation to meet this requirement?

Acme Corp is migrating its data pipeline to AWS and wants to ensure that all data is encrypted in transit to comply with regulatory requirements. They have a stringent security policy that mandates the use of IAM policies to enforce encryption settings on all data transfers. As the lead Data Engineer, you need to configure IAM policies to ensure all S3 buckets can only be accessed via HTTPS. Which of the following IAM policy statements would best meet these requirements?

John is a data engineer responsible for managing log data generated by an Amazon EMR cluster running various data processing workloads. The cluster generates a high volume of log data that needs to be securely stored and retrieved for audit purposes. John also needs to ensure that only specific team members have access to these logs and that all access is properly logged for further auditing. He has decided to use Amazon S3 as the storage solution for the logs and needs to make sure that the security policies are set correctly using AWS Identity and Access Management (IAM). Which combination of IAM policies and AWS services should John implement to achieve the above requirements?

You have recently joined a company as a data engineer. The company has a strict policy for data governance and data privacy. As part of the policy, all configuration changes within the AWS account need to be monitored and recorded meticulously. The company uses AWS Config for this purpose. Recently, several configuration changes were made to various AWS resources, and you need to verify if these changes comply with the company's policies. Which service or feature can you use to view the chronological order of configuration changes along with their details?