define_JADE_model(inputdir::String; run_file::String = "run")

This function reads a JADE run file in the Input/<inputdir> folder. In particular, this function reads the parameters that are needed in order to define the SDDP model.

Required Arguments

inputdir is the name of the subdirectory (within Input) that contains the JADE data files.

Keyword Arguments

run_file is the name of the csv file that contains the parameters we wish to load.

define_JADE_solve_options(inputdir::String; run_file = "run")

This function loads the training options for JADE.

Required Arguments

inputdir is the name of the subdirectory (within Input) where the run_file is located.

Keyword Arguments

run_file is the name of the csv file containing the parameters for the simulation.

define_JADE_simulation(inputdir::String; run_file = "run")

This function reads a csv file containing the settings for a JADE simulation.

Required Arguments

inputdir is the name of the subdirectory (within Input) where the run_file is located.

Keyword Arguments

run_file is the name of the csv file containing the parameters for the simulation.

define_JADE_model(;
    data_dir::String = "",
    policy_dir::String = "",
    start_year::Int = 0,
    start_week::Int = 1,
    total_weeks::Int = 52,
    hydro_sample_range::Union{Tuple{Int,Int},Int,UnitRange{Int},Vector} = 0,
    discount::Real = 0.0,
    losses::Symbol = :default,
    DIA::Int = 1,
    flow_penalties::Union{Tuple{Real,Real},NamedTuple{(:under, :over),<:Tuple{Real,Real}}} = (
        500.0,
        50.0,
    ),
    weekly_discounting::Bool = true,
    scale_reservoirs::Real = 1.0,
    scale_objective::Real = 1.0,
    use_terminal_mwvs::Bool = false,
    first_week_known::Bool = false,
    steady_state::Union{Nothing,Bool} = nothing,
    decision_rules::Vector{DecisionRule} = DecisionRule[],
    scenario_dir::String = "",
)

This function sets the parameters that are needed in order to define the SDDP model.

Required Arguments

data_dir is the name of the subdirectory (within Input) that contains the JADE data files.

policy_dir is the name of the subdirectory (within Output/data_dir) that the cuts and simulation output will be stored in.

start_year is the year that the policy is created for.

hydro_sample_range is the set of years that define the hydro inflow distribution.

Keyword Arguments

start_week is the first week of the start_year.

total_weeks is the total number of weeks that the policy is created for.

discount is the discount factor used for the steady-state model, must be less than 1. Setting this to 0 will set JADE to a finite horizon model. The terminal water values will only be used if discount is 0.

losses sets the loss model; this can be :none, :piecewise or :resistive.

DIA is the dynamic inflow adjustment parameter; 0 or 1 means no adjustment, higher values simulate an increased temporal correlation by increasing the variance.

flow_penalties is a Tuple of the cost of violating the lower bound, upper bound flow constraints, respectively.

weekly_discounting is a Bool which sets the discounting to occur in each stage rather than at the end of the horizon.

scale_reservoirs this may be increased to improve numerical stability (~100).

scale_objective scales down the objective function, this also scales down cuts (~1000).

use_terminal_mwvs if set to true this will use the MWVs stored in terminal_water_value.csv, if set to false these won't be used.

first_week_known if set to true the training will use deterministic inflows for the first week of the model. These will be sourced from the inflows recorded for that week and year.

steady_state if set to true the training will operate in steady-state mode, if false the model will use a finite-horizon.

decision_rules if any reservoir has restrictions on output, as a function of the storage, a DecisionRule can be applied. A vector of such decisions can be supplied via this argument.

define_JADE_solve_options(;
    iterations::Int = 0,
    random_seed::Int = Int(floor((time() - floor(time())) * 10000)),
    risk::Union{Tuple{Real,Real},NamedTuple{(:lambda, :beta),<:Tuple{Real,Real}}} = (
        0.0,
        1.0,
    ),
    warmstart_cuts::Bool = false,
    reset_starting_levels::Union{Symbol,Bool} = :default,
    saved_cuts::String = "",
    cutselection::Int = 1,
    fractionMC::Float64 = 1.0
    custom_inflow_file::String = ""
    loadcuts_if_nonempty::Bool = false,
    eoh_cutfile::String = "",
    write_eohcuts::Bool = false
)

This function defines the training options for JADE.

Keyword Arguments

iterations is the number of iterations of SDDP to complete when optimize_policy! is called.

random_seed sets the random seed prior to sampling for the forward pass.

risk is a Tuple (λ,β) defining the objective weight λ and the tail probability β.

warmstart_cuts if set to true, this will load stored cuts before training.

reset_starting_levels is used to reset the reservoirs in a steady-state model. This will enable a steady-state model to focus on a single starting set of reservoir levels (after initially finding an end-of-horizon value function).

saved_cuts is the policy subdirectory from which the cuts should be loaded. If not specified, cuts will be loaded from the policy subdirectory set for the model.

cutselection is an SDDP.jl parameter.

fractionMC is the proportion of forward passes that are sampled using Monte-Carlo; the rest are sequentially sampled from the sequences.

custom_inflow_file is the name of the file in the Input/data_dir directory containing the custom inflow sequences.

loadcuts_if_nonempty if set to true cuts will be loaded from file even if cuts are already in the model.

eoh_cutfile the name of end-of-horizon cut file for finite-horizon models.

write_eohcuts if set to true steady-state models will write end-of-horizon cuts files.

define_JADE_simulation(;
    sim_dir::String = "",
    sim_type::Symbol = :not_set,
    replications::Int = 0,
    sim_years = nothing,
    randomize_years::Bool = false,
    reset_starting_levels::Bool = false,
    number_of_cycles::Int = 1,
    initial_stage::Int = 1,
    initial_state::Union{Nothing,Dict{String,<:Real}} = nothing,
    random_seed::Int = Int(floor((time() - floor(time())) * 10000)),
)

This function defines the settings for a JADE simulation.

Keyword Arguments

sim_dir is the name of the subdirectory (within Output/<data_dir>/<policy_dir>) in which the simulation output will be stored.

sim_type is the type of simulation; either :monte_carlo or :historical.

replications is the number of sequences of inflows that are simulated. If the sim_type is :monte_carlo or randomize_years set to true then this argument must be set, otherwise it must not be set.

sim_years specifies the years that will be simulated.

randomize_years is used with historical simulations; if true this causes the simulation to randomly sample from sim_years, whereas if false the simulation will sample the years in sequence.

reset_starting_levels is used to reset the reservoirs in a steady-state model. This will enable a simulation to sample a number of inflows sequences, with the reservoirs starting at the from the same points for each sequence.

number_of_cycles is used with reset_starting_levels to show a simulation of multiple years, from a single single starting point. Must be used with a steady-state policy.

initial_state is a vector of initial storage levels for the simulation, indexed by the reservoirs.

initial_stage is the stage of the trained model that the simulation will start from.

random_seed sets the random seed prior to generating the random inflow sequences.

create_JADE_model(rundata::RunData, optimizer)

This function creates a JADEModel consisting of an SDDP policy graph, and the associated data.

Required Arguments

rundata is the RunData object containing all the SDDP model details.

optimizer is the JuMP optimizer you wish to use to train the policy.

function DecisionRule(
    slope::Real,
    intercept::Real,
    boundtype::Symbol,
    station::Symbol,
    reservoir::Symbol,
    flowtype::Symbol,
    weeks::T where {T<:Union{Tuple{Int,Int},Int,UnitRange{Int},Vector}} = 1:52,
)

This function constructs a DecisionRule object. See the JADE user documentation for further details. DecisionRules can be applied to spill and generation at hydro stations, these are both measured in MWh.

Required Arguments

slope is the slope, w.r.t. the current storage in reservoir, of the (linear) bound applied to generation and / or spill flow corresponding to station. Units are MWh / Mm³.

intercept is the bound applied to generation and / or spill flow corresponding to station when reservoir is at 0 storage. Units are (MWh).

boundtype is set to :upper, :lower, or :equality depending on how the flow should be restricted.

station is the hydro station that the decision rule restricts.

reservoir is the hydro reservoir whose affects the bound applied.

flowtype is set to :generation, :spill, or :combined.

weeks defines the weeks in the year that the decision rule is in effect. E.g. [1:10,43:52].

optimize_policy!(
    JADEmodel::JADEModel,
    solveoptions::JADESolveOptions;
    async::Bool = false,
    print_level::Int = 1,
)

This function trains the JADEmodel.

The output from this simulation will automatically be written into the Output/data_dir/policy_dir/sim_dir subdirectory.

Required Arguments

JADEmodel is the model detailing the year that we wish to create a policy for, and all the corresponding data.

solveoptions contains all the SDDP.jl settings, as well as settings and directories for loading cuts.

Keyword Arguments

async is a boolean that sets whether SDDP.jl will run in parallel. Currently recommended to be set to false.

print_level is an SDDP.jl setting.

simulate(JADEmodel::JADEModel, parameters::JADESimulation)

This function carries out a simulation of a JADE model; all the specifications for this simulation are defined within the parameters argument.

The output from this simulation will automatically be written into the Output/data_dir/policy_dir/sim_dir subdirectory.

Required Arguments

JADEmodel is the model detailing the scenario that we wish to simulate, and all the corresponding data.

parameters contains all the simulation information, including the number of replications, the type of simulation, the hydrological years to sample from, etc.

plot_storage(
    results::Array{Array{Dict{Symbol,Any},1},1},
    filename::String = randstring(6);
    reservoirs::Array{Symbol,1} = collect(keys(results[1][1][:reslevel])),
)

This function plots the storage in each of the reservoirs (Mm³) alongside the total storage (GWh) as an interactive webpage.

Required Arguments

results is the output from a JADE simulation.

Keyword Arguments

filename is the name of the file in which you wish to store the visualisation; the filename will be a random string if omitted.

reservoirs is an array of Symbols for the reservoirs that are to be visualised.

plot_prices(
    results::Array{Array{Dict{Symbol,Any},1},1},
    filename::String = randstring(6);
    nodes::Array{Symbol,1} = results[1][1][:prices].axes[1],
    blocks::Array{Symbol,1} = results[1][1][:prices].axes[2],
)

This function plots the prices for each of the nodes and load blocks alongside the total storage (GWh) as an interactive webpage.

Required Arguments

results is the output from a JADE simulation.

Keyword Arguments

nodes is an array of Symbols for the nodes that are to be visualised.

filename is the name of the file in which you wish to store the visualisation; the filename will be a random string if omitted.

blocks is an array of Symbols for the load blocks that are to be visualised.

gettimeseries(file::String)

Read timeseries with arbitrary number of columns from file.

Required Arguments

file is the name of the file from which JADE is loading Timeseries data.

Example File

% Data (A,B,C)
YEAR,WEEK,A,B,C
2005,1,4,18.68,4.49
2005,2,4,18.68,4.49

This function returns the year and week, given a stage.

For a data frame containing columns named WEEK and YEAR, return only the subset of data that falls within a relevant time range.

Determines if a time point is between two other time points, inclusive

parsefile(f::Function, file::String, trim::Bool = false)

This function reads in a data file and extracts each of the non-comment / non-empty rows, splits into into a vector using ',' as the delimiter, and then applies some custom processing f.

Required Arugments

f custom function that processes a line of the data file.

file path of the file.

Keyword Arguments

trim if true trailing commas will be discarded.

getitems(io::IOStream, trim::Bool)

This function takes an IOStream for a CSV file and reading the first line, decarding empty / comment rows, and otherwise splitting into into a vector using ',' as the delimiter. This vector is returned.

Required Arugments

io IOStream object for a CSV file.

trim if true trailing commas will be discarded.

parse_run_options(file::String)

This function loads a run file (CSV) creates a Dict that has keys corresponding to various JADE RunData fields, and their associated values.

Required Arugments

file is the path to the run file that is being loaded.

gethydros(file::String)

Description

Read list of hydro station data from file. Capacity in MW; Specific power in MW/cumec; Spillway max flow in cumecs (can be zero if no spillway exists, or "na" for unlimited spill). The columns MUST be ordered as shown below.

Example File

GENERATOR,HEAD_WATER_FROM,TAIL_WATER_TO,POWER_SYSTEM_NODE,CAPACITY,SPECIFIC_POWER,SPILLWAY_MAX_FLOW
Arapuni,Lake_Arapuni,Lake_Karapiro,NI,196.7,0.439847649,99999

Returns an inflow matrix (actually a vector) for a given year

getnaturalarcs(file::String)

Description

Read list of hydro station data from file. Canals, rivers, and other means of getting water from one place to another. Excludes power station turbines and spillways; those are covered in the hydrostations file. Min/max flow in cumecs. NA in MINFLOW converted to 0.0 NA in MAX_FLOW converted to 99999.0 The columns MUST be ordered as shown below.

Example File

ORIG,DEST,MIN_FLOW,MAX_FLOW
Lake_Wanaka,Lake_Dunstan, NA, NA
Lake_Hawea,Lake_Dunstan, 0.00, 99999

getterminalvalue(file::String)

Description

At the final stage terminal cost is a piecewise linear function of the national stored energy. We define a number of linear equations.

• Value of stored energy in hydro lakes at the end of the time horizon.
• STORED_ENERGY is the cumulative stored energy in GWh.
• VALUE is the marginal value in /MWh. This column should be a decreasing sequence.

The columns MUST be ordered as shown below.

Example File

STORED_ENERGY, VALUE
1000, 137.218398630355   % first 1000 GWh is worth ~ $137/MWh
1500, 85.9718321526058

getthermalstations(file::String, nodes::Vector{Symbol})

Description

Read thermal station list from file. The columns MUST be ordered as shown below.

Example File

% Thermal station data,,,,,,,,,
% Heat rate in GJ/MWh,,,,,,,,,
% Capacity in MW,,,,,,,,,
% O&M cost in $/MWh,,,,,,,,,
GENERATOR,NODE,FUEL,HEAT_RATE,CAPACITY,OMCOST,START_YEAR,START_WEEK,END_YEAR,END_WEEK
Stratford_220KV,NI,gas,7.6,377,0,0,0,0,0
Huntly_e3p,NI,gas,7.2,403,0,2007,23,0,0
Huntly_main_g1,NI,coal,10.3,250,0,0,0,0,0
Huntly_main_g2,NI,coal,10.3,250,0,0,0,0,0

initialisereservoirs(file::String, limits::String)

Description

Read list of reservoirs from file. Capacity and initial contents in Mm³. The columns MUST be ordered as shown below.

Example File

RESERVOIR, INFLOW_REGION, CAPACITY, INI_STATE
Lake_Benmore, SI, 423451075.96799916, 322000320.233399

getdemand(file::String, durations::JADE.TimeSeries{Dict{Symbol,Float64}})

Read the demand in MW

Example

% Demand (MW):
NODE,YEAR,WEEK,peak,shoulder,offpeak
NI,2005,1,2132.15,1616.51,1116.58

This function checks that there are no outages greater than the capacity of a station and outage data spans run length

get_file_directory(x::String, rundata::RunData; verbose::Bool = true)

This function looks in various locations for input file x. If the rundata references a scenario_dir then the first directory to search is <datadir>/datafiles/<scenario_dir>.

Required Arguments

x is the name of the input file. rundata is a JADE RunData object.

Keyword Arguments

verbose if set to true the function will print info to the REPL.

check_settings_compatibility(;
    rundata::Union{Nothing,RunData} = nothing,
    solveoptions::Union{Nothing,JADESolveOptions} = nothing,
    simulation::Union{Nothing,JADESimulation} = nothing,
)

This function checks the JADE settings for compatibility / consistency.

Keyword Arguments

rundata contains the JADE model options. solveoptions contains the JADE training options. simulation contains the JADE simulation settings.

out_neighbors(vertex::Symbol, edges::Vector{NTuple{2,Symbol}})

Helper function to work out nodes that are adjacent and downstream.

Required Arguments

vertex is the vertex in the network we are finding downstream neighbors for.

edges is a vector of all the edges in the hydro network.

hasdownstream(
    sets::Sets,
    station_arcs::Dict{NTuple{2,Symbol},StationArc},
)

This function is used to determine which hydro stations are present downstream from all reservoirs.

Inputs

sets A JADE set structure. Should include all arcs and nodes. station_arcs A dictionary we will use to get the station name from a station arc.

Returns

reservoirhasdownstream A dictionary indexed by reservoirs that stores a list of hydro stations that each reservoir has between itself and the sea.

adjustinflows(inflow_file::String, rundata::RunData)

Description

This method applies DIA to the inflow file.

diatofile(
    adjusted::Dict{Symbol,TimeSeries{Float64}},
    outpath::String,
    policy_dir::String,
)

Description

This function saves DIA adjusted inflows to file.

Required Arguments

adjusted is the dictionary of DIA-adjusted inflows outpath is the filename for adjusted inflows policy_dir is the subdirectory where the outputs are stored

write_sim_results(results, data::JADEData, parameters::JADESimulation)

This function writes results from a simulation.

Required arguments

results is a dictionary of results.

data is a JADE data object for the problem simulated.

parameters contains all the simulation settings.

This function saves output from policy generation.

An object containing the JADE model settings, required to define the stage problems in SDDP.jl.

Fields

data_dir::String directory for writing output.

start_yr::Int first year in the problem.

start_wk::Int first week in the problem.

number_of_wks::Int the number of weeks that we solve for.

sample_years::Vector{Int} array of years that inflows are sampled from.

dialength::Int inflow correlation length.

penalty_lb::Float64 penalty for flows below lower bound.

penalty_ub::Float64 penalty for flows above upper bound.

policy_dir::String output directory for that the policy cuts and settings are stored in.

discount::Float64 steady-state discount factor (0 if finite-horizon).

losses::Symbol type of losses (:none,:piecewise,:resistive).

nscenarios::Int number of possible inflow realisations per week.

weekly_discounting::Bool set to true if (equivalent) discount should applied weekly.

scale_reservoirs::Float64 scale factor for reservoirs.

scale_objective::Float64 scale factor for objective.

use_terminal_mwvs::Bool boolean specifying whether we use the given terminal MWVs.

first_week_known::Bool set to true if first-week inflow is set to historical value from training year.

steady_state::Bool set to true if the model is steady-state.

decision_rules::Vector{DecisionRule} vector of decision rules that can restrict how reservoirs are operated.

scenario_dir::String name of directory within Input/<datadir>/datafiles/ containing data files that change within the scenario.

An object containing SDDP settings used when training a JADE model.

Fields

iterations::Int sddp iterations to be performed.

riskmeasure::SDDP.AbstractRiskMeasure sddp risk measure to use (set up to be CVaR).

warmstart_cuts::Bool if true will load existing cuts, either from local directory or 'savedcuts'.

reset_starting_levels::Union{Symbol,Bool} if true will reset starting storage levels to initial values each forward pass.

seed::Int random seed.

savedcuts::String alternate folder from which to retrieve cuts.

cutselection::Int sddp cut_deletion_minimum parameter.

fractionMC::Float64 fraction of forward passes using monte-carlo.

custom_inflow_file::String file specifying sequences of inflows to be used in forward pass (fractionMC must be < 1).

loadcuts_if_nonempty::Bool set this to true if the cut file should be loaded into a model with existing cuts.

eoh_cutfile::String EOH cut file to load as the cuts for the final stage of a finite-horizon model.

write_eohcuts::Bool if true a steady-state model will write a set of files to Input/<data_dor>/EOH.

An object containing JADE simulation settings.

Fields

sim_dir::String is the folder that the simulation output should be written to (Output/<data_dir>/<policy_dir>/<sim_dir>).

sim_type::Symbol is the type of simulation, either :monte_carlo or :historical.

replications::Int is the number of replications of the simulation.

sim_years::Union{Nothing,Vector{Int}} is either nothing or a vector of years that should be sampled from in the simulation.

randomize_years::Bool if true the simulation will randomly choose years from sim_years (with replacement); otherwise the simulation sequentially samples the years from sim_years (in this case replications is automatically set).

reset_starting_levels::Union{Symbol,Bool} if true the simulation will reset the initial storage levels after each iteration, even for steady-state models.

number_of_cycles::Int for steady-state models, if this is greater than 1, each replication of the simulation will run number_of_cycles times; reset_starting_levels must be set to true.

initial_stage::Int is the stage (not the week) of the simulation that the simulation will start from.

initial_state::Union{Nothing,Dict{String,Float64}} is the initial state variables in the initial stage

random_seed::Int is the random seed for the simulation.

An object containing all the data required to run the JADE model.

Fields

rundata Run-related constants.

thermal_stations Dictionary of thermal power station properties.

hydro_stations Dictionary of hydro station properties.

reservoirs Dictionary of reservoir properties.

fuel_costs Time series of weekly fuel costs.

carbon_content Carbon content for each fuel type.

inflow_mat Structure for storing inflow scenarios.

station_arcs Properties of arcs that correspond to hydro releases or spills.

natural_arcs Arcs not going through hydro stations.

nodehas Dictionary of node properties.

spMax The maximum specific power value.

transmission Dictionary of transmission line properties.

loops Array of independent cycles in the transmission network, each entry being an array of arcs.

durations Timeseries with block durations.

demand Demand data.

outage Outage each week for each station.

dr_tranches Tranches for power-based demand response and load shedding.

en_tranches Tranches for energy-based demand response.

terminal_eqns Equations for terminal water value.

sets Structure containing all sets for the JADE model.

function read_finalcuts_from_file(
	model::SDDP.PolicyGraph{T},
	filename::String,
	cutsfromstage::Int,
	laststage::Int;
	node_name_parser::Function = SDDP._node_name_parser,
) where {T}

Required Arguments

model The SDDP PolicyGraph that cuts are being loaded into.

filename the full path to the cuts file being read.

cutsfromstage the index of the stage (not necessarily the week of the year) that cuts should be extracted from.

laststage the index of the stage of the model that the cuts are being loaded into.

node_name_parser SDDP function to convert the Int for laststage into the name of the SDDP node.

function write_cuts_to_file(
	model::SDDP.PolicyGraph{T},
	filename::String,
	start_wk::Int,
) where {T}

This function writes a cuts json file that is modified so that stage 1 is always associated with week 1. This can only be used with a 52-week model. (It should only be used with steady-state JADE models.)

Required Arguments

model The SDDP PolicyGraph that cuts are being written from.

filename the full path to the cuts file being written.

start_wk the week of the year that corresponds to stage 1 in the SDDP model.

WrapHistorical(
    scenarios::Vector{Vector{Tuple{T,S}}},
    probability::Vector{Float64},
) where {T,S}

A sampling scheme that samples a scenario from the vector of scenarios scenarios according to probability.

Example

WrapHistorical(
    [
        [(1, 0.5), (2, 1.0), (3, 0.5)],
        [(1, 0.5), (2, 0.0), (3, 1.0)],
        [(1, 1.0), (2, 0.0), (3, 0.0)]
    ],
    [0.2, 0.5, 0.3],
)

WrapHistorical(scenarios::Vector{Vector{Tuple{T,S}}}) where {T,S}

A deterministic sampling scheme that iterates through the vector of provided scenarios. The initial states in each iteration (other than the first) are set to the ending state of the previous forward pass.

Used in the forward pass of the training to enable custom inflow sequences in steady-state models (see solve.jl).

Example

WrapHistorical([
    [(1, 0.5), (2, 1.0), (3, 0.5)],
    [(1, 0.5), (2, 0.0), (3, 1.0)],
    [(1, 1.0), (2, 0.0), (3, 0.0)],
])

function InSampleMonteCarlo2(;
	max_depth::Int = 0,
	initial_stage::Int = 1,
	terminate_on_cycle::Bool = false,
	terminate_on_dummy_leaf::Bool = true,
	rollout_limit::Function = i -> typemax(Int),
)

This function creates an instance of the InSampleMonteCarlo2 sampling_scheme. This is used for Monte Carlo simulations that may start from an arbitrary stage (initial_stage).

Keyword Arguments

max_depth is the maximum number of stages to simulate.

initial_stage is the stage number of the initial stage.

terminate_on_cycle if true the simulation will stop if it returns to a previously visited node.

terminate_on_dummy_leaf if true the simulation will may stop if it the probabilities of the noises don't sum to 1.

rollout_limit is a function that allows the number of simulated stages to vary with the training iteration number.