Trino Optimizer
by treasure-data
TD Trino performance optimization including CTAS (5x faster), UDP bucketing for ID lookups, magic comments for join distribution, REGEXP_LIKE vs LIKE, and approx functions.
Skill Details
Repository Files
1 file in this skill directory
name: trino-optimizer description: TD Trino performance optimization including CTAS (5x faster), UDP bucketing for ID lookups, magic comments for join distribution, REGEXP_LIKE vs LIKE, and approx functions.
TD Trino Query Optimizer
Output Optimization
CTAS is 5x faster than SELECT (skips JSON serialization):
create table results as
select td_time_string(time, 'd!', 'JST') as date, count(*) as events
from events
where td_interval(time, '-1M', 'JST')
group by 1
User-Defined Partitioning (UDP)
Hash partition for fast ID lookups on large tables (>100M rows):
create table customer_events with (
bucketed_on = array['customer_id'],
bucket_count = 512
) as
select * from raw_events
where td_interval(time, '-30d', 'JST')
Accelerated queries (equality on all bucketing columns):
select * from customer_events
where customer_id = 12345
and td_interval(time, '-7d', 'JST')
NOT accelerated (missing bucketing column):
select * from customer_events
where td_interval(time, '-7d', 'JST')
UDP for colocated joins:
-- set session join_distribution_type = 'PARTITIONED'
-- set session colocated_join = 'true'
select a.*, b.*
from customer_events_a a
join customer_events_b b on a.customer_id = b.customer_id
Magic Comments for Join Distribution
-- BROADCAST: Small right table fits in memory
-- set session join_distribution_type = 'BROADCAST'
select * from large_table, small_lookup
where large_table.id = small_lookup.id
-- PARTITIONED: Both tables large or memory issues
-- set session join_distribution_type = 'PARTITIONED'
select * from large_table_a, large_table_b
where large_table_a.id = large_table_b.id
REGEXP_LIKE vs Multiple LIKE
-- BAD: Multiple LIKE clauses
where column like '%android%' or column like '%ios%' or column like '%mobile%'
-- GOOD: Single REGEXP_LIKE
where regexp_like(column, 'android|ios|mobile')
Approximate Functions
approx_distinct(user_id) -- vs count(distinct user_id)
approx_percentile(response_time, 0.95)
approx_set(column) -- HyperLogLog sketch
~2% error rate, dramatically reduced memory.
Common Errors
| Error | Fix |
|---|---|
| Query exceeded memory limit | Narrow time range, use approx functions, try PARTITIONED join |
| PAGE_TRANSPORT_TIMEOUT | Reduce columns, use CTAS, process in smaller chunks |
| Query timeout | Add time filters, use limit for testing |
Optimization Checklist
- Time filter (td_interval/td_time_range)
- Specific columns (not select *)
- approx_distinct for unique counts
- regexp_like for pattern matching
- CTAS for large results
- UDP for frequent ID lookups
Related Skills
Attack Tree Construction
Build comprehensive attack trees to visualize threat paths. Use when mapping attack scenarios, identifying defense gaps, or communicating security risks to stakeholders.
Grafana Dashboards
Create and manage production Grafana dashboards for real-time visualization of system and application metrics. Use when building monitoring dashboards, visualizing metrics, or creating operational observability interfaces.
Matplotlib
Foundational plotting library. Create line plots, scatter, bar, histograms, heatmaps, 3D, subplots, export PNG/PDF/SVG, for scientific visualization and publication figures.
Scientific Visualization
Create publication figures with matplotlib/seaborn/plotly. Multi-panel layouts, error bars, significance markers, colorblind-safe, export PDF/EPS/TIFF, for journal-ready scientific plots.
Seaborn
Statistical visualization. Scatter, box, violin, heatmaps, pair plots, regression, correlation matrices, KDE, faceted plots, for exploratory analysis and publication figures.
Shap
Model interpretability and explainability using SHAP (SHapley Additive exPlanations). Use this skill when explaining machine learning model predictions, computing feature importance, generating SHAP plots (waterfall, beeswarm, bar, scatter, force, heatmap), debugging models, analyzing model bias or fairness, comparing models, or implementing explainable AI. Works with tree-based models (XGBoost, LightGBM, Random Forest), deep learning (TensorFlow, PyTorch), linear models, and any black-box model
Pydeseq2
Differential gene expression analysis (Python DESeq2). Identify DE genes from bulk RNA-seq counts, Wald tests, FDR correction, volcano/MA plots, for RNA-seq analysis.
Query Writing
For writing and executing SQL queries - from simple single-table queries to complex multi-table JOINs and aggregations
Pydeseq2
Differential gene expression analysis (Python DESeq2). Identify DE genes from bulk RNA-seq counts, Wald tests, FDR correction, volcano/MA plots, for RNA-seq analysis.
Scientific Visualization
Meta-skill for publication-ready figures. Use when creating journal submission figures requiring multi-panel layouts, significance annotations, error bars, colorblind-safe palettes, and specific journal formatting (Nature, Science, Cell). Orchestrates matplotlib/seaborn/plotly with publication styles. For quick exploration use seaborn or plotly directly.
