pyspark.sql.functions.hll_union#

pyspark.sql.functions.hll_union(col1, col2, allowDifferentLgConfigK=None)[source]#

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is unset or set to false.

New in version 3.5.0.

Parameters

col1Column or column name
col2Column or column name
allowDifferentLgConfigKbool, optional: Allow sketches with different lgConfigK values to be merged (defaults to false).

Returns

Column: The binary representation of the merged HllSketch.

See also

pyspark.sql.functions.hll_union_agg()
pyspark.sql.functions.hll_sketch_agg()
pyspark.sql.functions.hll_sketch_estimate()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1,4),(2,5),(2,5),(3,6)], "struct<v1:int,v2:int>")
>>> df = df.agg(
...     sf.hll_sketch_agg("v1").alias("sketch1"),
...     sf.hll_sketch_agg("v2").alias("sketch2")
... )
>>> df.select(sf.hll_sketch_estimate(sf.hll_union(df.sketch1, "sketch2"))).show()
+-------------------------------------------------------+
|hll_sketch_estimate(hll_union(sketch1, sketch2, false))|
+-------------------------------------------------------+
|                                                      6|
+-------------------------------------------------------+