Vojtech Tuma
1 min readDec 26, 2018

--

Losing Constant Folding for UDFs is not really a case, right? Because you can imitate the same effect: instead of typing “dataFrame.select(myUdf + 1)”, you move the +1 into the body of myUdf — which may even make more sense logically

As for predicate pushdown — that can be a really beneficial boost, but applies only to filters. So not at all for UDAFs or UDFs which just compute or construct something. For those, Ser/De may be a big issue, though

--

--

Vojtech Tuma
Vojtech Tuma

Written by Vojtech Tuma

#books - #running - #pullups - #boardGames - #dataScience - #programming - #trolling - #etc

No responses yet