Databricks Certified Associate-Developer for Apache Spark 3.5 Exam Free Dumps

dumps2025-10-29T10:00:47+08:00

dumps Databricks Databricks Certified Associate-Developer for Apache Spark 3.5 exam, Databricks Certified Associate-Developer for Apache Spark 3.5 exam dumps

Databricks Certified Associate-Developer for Apache Spark 3.5 exam is one for Apache Spark Developer Associate certification, which assesses the understanding of the Apache Spark Architecture and Components and the ability to apply the Spark DataFrame API to complete basic data manipulation tasks within a Spark session. There are 45 multiple-choice questions in real Databricks Certified Associate-Developer for Apache Spark 3.5 exam, and you have 90 minutes to answer all the questions. The available language is English. The registration fee is $200. We provide the latest Databricks Certified Associate-Developer for Apache Spark 3.5 exam dumps, which can ensure you pass the test easily. Free Databricks Certified Associate-Developer for Apache Spark 3.5 exam dumps are available below.

Page 1 of 5

1. An MLOps engineer is building a Pandas UDF that applies a language model that translates English strings into Spanish. The initial code is loading the model on every call to the UDF, which is hurting the performance of the data pipeline.

The initial code is:

def in_spanish_inner(df: pd.Series) -> pd.Series:

model = get_translation_model(target_lang='es')

return df.apply(model)

in_spanish = sf.pandas_udf(in_spanish_inner, StringType())

How can the MLOps engineer change this code to reduce how many times the language model is loaded?

Convert the Pandas UDF to a PySpark UDF

Convert the Pandas UDF from a Series → Series UDF to a Series → Scalar UDF

Run the in_spanish_inner() function in a mapInPandas() function call

Convert the Pandas UDF from a Series → Series UDF to an Iterator[Series] → Iterator[Series] UDF

2. A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame is too large to fit entirely in memory.

What is the likely behavior when Spark runs out of memory to store the DataFrame?

Spark duplicates the DataFrame in both memory and disk. If it doesn't fit in memory, the DataFrame is stored and retrieved from the disk entirely.

Spark splits the DataFrame evenly between memory and disk, ensuring balanced storage utilization.

Spark will store as much data as possible in memory and spill the rest to disk when memory is full, continuing processing with performance overhead.

Spark stores the frequently accessed rows in memory and less frequently accessed rows on disk, utilizing both resources to offer balanced performance.

3. A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off.

How can this be achieved?

By configuring the option checkpointLocation during readStream

By configuring the option recoveryLocation during the SparkSession initialization

By configuring the option recoveryLocation during writeStream

By configuring the option checkpointLocation during writeStream

4. A data scientist is analyzing a large dataset and has written a PySpark script that includes several transformations and actions on a DataFrame. The script ends with a collect() action to retrieve the results.

How does Apache Spark™'s execution hierarchy process the operations when the data scientist runs this script?

The script is first divided into multiple applications, then each application is split into jobs, stages, and finally tasks.

The entire script is treated as a single job, which is then divided into multiple stages, and each stage is further divided into tasks based on data partitions.

The collect() action triggers a job, which is divided into stages at shuffle boundaries, and each stage is split into tasks that operate on individual data partitions.

Spark creates a single task for each transformation and action in the script, and these tasks are grouped into stages and jobs based on their dependencies.

5. A developer is trying to join two tables, sales.purchases_fct and sales.customer_dim, using the following code:

fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'))

The developer has discovered that customers in the purchases_fct table that do not exist in the customer_dim table are being dropped from the joined table.

Which change should be made to the code to stop these customer records from being dropped?

fact_df = purch_df.join(cust_df,

col('customer_id') ==

col('custid'), 'left')

fact_df = cust_df.join(purch_df,

col('customer_id') ==

col('custid'))

fact_df = purch_df.join(cust_df,

col('cust_id') ==

col('customer_id'))

fact_df = purch_df.join(cust_df,

col('customer_id') ==

col('custid'), 'right_outer')

6. A data engineer is reviewing a Spark application that applies several transformations to a DataFrame but notices that the job does not start executing immediately.

Which two characteristics of Apache Spark's execution model explain this behavior? Choose 2 answers:

The Spark engine requires manual intervention to start executing transformations.

Only actions trigger the execution of the transformation pipeline.

Transformations are executed immediately to build the lineage graph.

The Spark engine optimizes the execution plan during the transformations, causing delays.

Transformations are evaluated lazily.

7. A developer needs to produce a Python dictionary using data stored in a small Parquet table, which looks like this:

The resulting Python dictionary must contain a mapping of region -> region id containing the smallest 3 region_id values.

Which code fragment meets the requirements?

A)

Option A

Option B

Option C

Option D

8. An engineer has a large ORC file located at /file/test_data.orc and wants to read only specific columns to reduce memory usage.

Which code fragment will select the columns, i.e., col1, col2, during the reading process?

spark.read.orc("/file/test_data.orc").filter("col1 = 'value' ").select("col2")

spark.read.format("orc").select("col1", "col2").load("/file/test_data.orc")

spark.read.orc("/file/test_data.orc").selected("col1", "col2")

spark.read.format("orc").load("/file/test_data.orc").select("col1", "col2")

9. Given the code fragment:

import pyspark.pandas as ps

psdf = ps.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

Which method is used to convert a Pandas API on Spark DataFrame (pyspark.pandas.DataFrame) into a standard PySpark DataFrame (pyspark.sql.DataFrame)?

psdf.to_spark()

psdf.to_pyspark()

psdf.to_pandas()

psdf.to_dataframe()

10. A Spark engineer is troubleshooting a Spark application that has been encountering out-of-memory errors during execution. By reviewing the Spark driver logs, the engineer notices multiple "GC overhead limit exceeded" messages.

Which action should the engineer take to resolve this issue?

Optimize the data processing logic by repartitioning the DataFrame.

Modify the Spark configuration to disable garbage collection

Increase the memory allocated to the Spark Driver.

Cache large DataFrames to persist them in memory.

Page 2 of 5

11. A DataFrame df has columns name, age, and salary. The developer needs to sort the DataFrame by age in ascending order and salary in descending order.

Which code snippet meets the requirement of the developer?

df.orderBy(col("age").asc(), col("salary").asc()).show()

df.sort("age", "salary", ascending=[True, True]).show()

df.sort("age", "salary", ascending=[False, True]).show()

df.orderBy("age", "salary", ascending=[True, False]).show()

12. What is the difference between df.cache() and df.persist() in Spark DataFrame?

Both cache() and persist() can be used to set the default storage level (MEMORY_AND_DISK_SER)

Both functions perform the same operation. The persist() function provides improved performance as its default storage level is DISK_ONL

persist() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK_SER) and cache() - Can be used to set different storage levels to persist the contents of the DataFrame.

cache() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK) and persist() - Can be used to set different storage levels to persist the contents of the DataFrame

13. A data analyst builds a Spark application to analyze finance data and performs the following operations: filter, select, groupBy, and coalesce.

Which operation results in a shuffle?

groupBy

filter

select

coalesce

14. A data engineer is asked to build an ingestion pipeline for a set of Parquet files delivered by an upstream team on a nightly basis. The data is stored in a directory structure with a base path of "/path/events/data". The upstream team drops daily data into the underlying subdirectories following the convention year/month/day.

A few examples of the directory structure are:

Which of the following code snippets will read all the data within the directory structure?

df = spark.read.option("inferSchema", "true").parquet("/path/events/data/")

df = spark.read.option("recursiveFileLookup", "true").parquet("/path/events/data/")

df = spark.read.parquet("/path/events/data/*")

df = spark.read.parquet("/path/events/data/")

15. Given a DataFrame df that has 10 partitions, after running the code:

result = df.coalesce(20)

How many partitions will the result DataFrame have?

Same number as the cluster executors

16. Given the following code snippet in my_spark_app.py:

What is the role of the driver node?

The driver node orchestrates the execution by transforming actions into tasks and distributing them to worker nodes

The driver node only provides the user interface for monitoring the application

The driver node holds the DataFrame data and performs all computations locally

The driver node stores the final result after computations are completed by worker nodes

17. A Spark developer wants to improve the performance of an existing PySpark UDF that runs a hash function that is not available in the standard Spark functions library.

The existing UDF code is:

import hashlib

import pyspark.sql.functions as sf

from pyspark.sql.types import StringType

def shake_256(raw):

return hashlib.shake_256(raw.encode()).hexdigest(20)

shake_256_udf = sf.udf(shake_256, StringType())

The developer wants to replace this existing UDF with a Pandas UDF to improve performance. The developer changes the definition of shake_256_udf to this:CopyEdit

shake_256_udf = sf.pandas_udf(shake_256, StringType())

However, the developer receives the error:

What should the signature of the shake_256() function be changed to in order to fix this error?

def shake_256(df: pd.Series) -> str:

def shake_256(df: Iterator[pd.Series]) -> Iterator[pd.Series]:

def shake_256(raw: str) -> str:

def shake_256(df: pd.Series) -> pd.Series:

18. A developer is working with a pandas DataFrame containing user behavior data from a web application.

Which approach should be used for executing a groupBy operation in parallel across all workers in Apache Spark 3.5?

A) Use the applylnPandas API

B)

Option A

Option B

Option C

Option D

19. Given:

python

CopyEdit

spark.sparkContext.setLogLevel("<LOG_LEVEL>")

Which set contains the suitable configuration settings for Spark driver LOG_LEVELs?

ALL, DEBUG, FAIL, INFO

ERROR, WARN, TRACE, OFF

WARN, NONE, ERROR, FATAL

FATAL, NONE, INFO, DEBUG

20. An engineer wants to join two DataFrames df1 and df2 on the respective employee_id and emp_id columns:

df1: employee_id INT, name STRING

df2: emp_id INT, department STRING

The engineer uses:

result = df1.join(df2, df1.employee_id == df2.emp_id, how='inner')

What is the behaviour of the code snippet?

The code fails to execute because the column names employee_id and emp_id do not match automatically

The code fails to execute because it must use on='employee_id' to specify the join column explicitly

The code fails to execute because PySpark does not support joining DataFrames with a different structure

The code works as expected because the join condition explicitly matches employee_id from df1 with emp_id from df2

Page 3 of 5

21. A data engineer has been asked to produce a Parquet table which is overwritten every day with the latest data. The downstream consumer of this Parquet table has a hard requirement that the data in this table is produced with all records sorted by the market_time field.

Which line of Spark code will produce a Parquet table that meets these requirements?

final_df .sort("market_time") .write .format("parquet") .mode("overwrite") .saveAsTable("output.market_events")

final_df .orderBy("market_time") .write .format("parquet") .mode("overwrite") .saveAsTable("output.market_events")

final_df .sort("market_time") .coalesce(1) .write .format("parquet") .mode("overwrite") .saveAsTable("output.market_events")

final_df .sortWithinPartitions("market_time") .write .format("parquet") .mode("overwrite") .saveAsTable("output.market_events")

22. In the code block below, aggDF contains aggregations on a streaming DataFrame:

Which output mode at line 3 ensures that the entire result table is written to the console during each trigger execution?

complete

append

replace

aggregate

23. A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed storage. The cluster has 10 nodes, each with 16 CPUs.

Spark UI shows:

Low number of Active Tasks

Many tasks complete in milliseconds

Fewer tasks than available CPUs

Which approach should be used to adjust the partitioning for optimal resource allocation?

Set the number of partitions equal to the total number of CPUs in the cluster

Set the number of partitions to a fixed value, such as 200

Set the number of partitions equal to the number of nodes in the cluster

Set the number of partitions by dividing the dataset size (1 TB) by a reasonable partition size, such as 128 MB

24. What is the behavior for function date_sub(start, days) if a negative value is passed into the days parameter?

The same start date will be returned

An error message of an invalid parameter will be returned

The number of days specified will be added to the start date

The number of days specified will be removed from the start date

25. A data engineer is building an Apache Spark™ Structured Streaming application to process a stream of JSON events in real time. The engineer wants the application to be fault-tolerant and resume processing from the last successfully processed record in case of a failure. To achieve this, the data engineer decides to implement checkpoints.

Which code snippet should the data engineer use?

query = streaming_df.writeStream .format("console") .option("checkpoint", "/path/to/checkpoint") .outputMode("append") .start()

query = streaming_df.writeStream .format("console") .outputMode("append") .option("checkpointLocation", "/path/to/checkpoint") .start()

query = streaming_df.writeStream .format("console") .outputMode("complete") .start()

query = streaming_df.writeStream .format("console") .outputMode("append") .start()

26. A data engineer is running a batch processing job on a Spark cluster with the following configuration:

10 worker nodes

16 CPU cores per worker node

64 GB RAM per node

The data engineer wants to allocate four executors per node, each executor using four cores.

What is the total number of CPU cores used by the application?

160

27. A developer is running Spark SQL queries and notices underutilization of resources. Executors are idle, and the number of tasks per stage is low.

What should the developer do to improve cluster utilization?

Increase the value of spark.sql.shuffle.partitions

Reduce the value of spark.sql.shuffle.partitions

Increase the size of the dataset to create more partitions

Enable dynamic resource allocation to scale resources as needed

28. What is the risk associated with this operation when converting a large Pandas API on Spark DataFrame back to a Pandas DataFrame?

The conversion will automatically distribute the data across worker nodes

The operation will fail if the Pandas DataFrame exceeds 1000 rows

Data will be lost during conversion

The operation will load all data into the driver's memory, potentially causing memory overflow

29. A developer notices that all the post-shuffle partitions in a dataset are smaller than the value set for spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold.

Which type of join will Adaptive Query Execution (AQE) choose in this case?

A Cartesian join

A shuffled hash join

A broadcast nested loop join

A sort-merge join

30. Which configuration can be enabled to optimize the conversion between Pandas and PySpark DataFrames using Apache Arrow?

spark.conf.set("spark.pandas.arrow.enabled", "true")

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

spark.conf.set("spark.sql.execution.arrow.enabled", "true")

spark.conf.set("spark.sql.arrow.pandas.enabled", "true")

Page 4 of 5

31. A data engineer writes the following code to join two DataFrames df1 and df2:

df1 = spark.read.csv("sales_data.csv") # ~10 GB

df2 = spark.read.csv("product_data.csv") # ~8 MB

result = df1.join(df2, df1.product_id == df2.product_id)

Which join strategy will Spark use?

Shuffle join, because AQE is not enabled, and Spark uses a static query plan

Broadcast join, as df2 is smaller than the default broadcast threshold

Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently

Shuffle join because no broadcast hints were provided

32. A developer wants to test Spark Connect with an existing Spark application.

What are the two alternative ways the developer can start a local Spark Connect server without changing their existing application code? (Choose 2 answers)

Execute their pyspark shell with the option --remote "https://localhost"

Execute their pyspark shell with the option --remote "sc://localhost"

Set the environment variable SPARK_REMOTE="sc://localhost" before starting the pyspark shell

Add .remote("sc://localhost") to their SparkSession.builder calls in their Spark code

Ensure the Spark property spark.connect.grpc.binding.port is set to 15002 in the application code

33. A Spark developer is building an app to monitor task performance. They need to track the maximum task processing time per worker node and consolidate it on the driver for analysis.

Which technique should be used?

Use an RDD action like reduce() to compute the maximum time

Use an accumulator to record the maximum time on the driver

Broadcast a variable to share the maximum time among workers

Configure the Spark UI to automatically collect maximum times

34. A Spark application developer wants to identify which operations cause shuffling, leading to a new stage in the Spark execution plan.

Which operation results in a shuffle and a new stage?

DataFrame.groupBy().agg()

DataFrame.filter()

DataFrame.withColumn()

DataFrame.select()

35. Which UDF implementation calculates the length of strings in a Spark DataFrame?

df.withColumn("length", spark.udf("len", StringType()))

df.select(length(col("stringColumn")).alias("length"))

spark.udf.register("stringLength", lambda s: len(s))

df.withColumn("length", udf(lambda s: len(s), StringType()))

36. A data engineer observes that an upstream streaming source sends duplicate records, where duplicates share the same key and have at most a 30-minute difference in event_timestamp.

The engineer adds:

dropDuplicatesWithinWatermark("event_timestamp", "30 minutes")

What is the result?

It is not able to handle deduplication in this scenario

It removes duplicates that arrive within the 30-minute window specified by the watermark

It removes all duplicates regardless of when they arrive

It accepts watermarks in seconds and the code results in an error

37. A Data Analyst needs to retrieve employees with 5 or more years of tenure.

Which code snippet filters and shows the list?

employees_df.filter(employees_df.tenure >= 5).show()

employees_df.where(employees_df.tenure >= 5)

filter(employees_df.tenure >= 5)

employees_df.filter(employees_df.tenure >= 5).collect()

38. Given a CSV file with the content:

And the following code:

from pyspark.sql.types import *

schema = StructType([

StructField("name", StringType()),

StructField("age", IntegerType())

])

spark.read.schema(schema).csv(path).collect()

What is the resulting output?

[Row(name='bambi'), Row(name='alladin', age=20)]

[Row(name='alladin', age=20)]

[Row(name='bambi', age=None), Row(name='alladin', age=20)]

The code throws an error due to a schema mismatch.

39. A Spark application is experiencing performance issues in client mode because the driver is resource-constrained.

How should this issue be resolved?

Add more executor instances to the cluster

Increase the driver memory on the client machine

Switch the deployment mode to cluster mode

Switch the deployment mode to local mode

40. A data scientist of an e-commerce company is working with user data obtained from its subscriber database and has stored the data in a DataFrame df_user. Before further processing the data, the data scientist wants to create another DataFrame df_user_non_pii and store only the non-PII columns in this DataFrame. The PII columns in df_user are first_name, last_name, email, and birthdate.

Which code snippet can be used to meet this requirement?

A. df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

B. df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

C. df_user_non_pii = df_user.dropfields("first_name", "last_name", "email", "birthdate")

D. df_user_non_pii = df_user.dropfields("first_name, last_name, email, birthdate")

Page 5 of 5

41. A data engineer is working on a Streaming DataFrame streaming_df with the given streaming data:

Which operation is supported with streamingdf ?

A. streaming_df. select (countDistinct ("Name") )

B. streaming_df.groupby("Id") .count ()

C. streaming_df.orderBy("timestamp").limit(4)

D. streaming_df.filter (col("count") < 30).show()

Databricks Certified Associate-Developer for Apache Spark 3.5 Exam Free Dumps

Databricks Certified Associate-Developer for Apache Spark 3.5 Exam Free Dumps

Share this post

Related Posts