基于TD3的电动汽车复合电源能量管理策略研究

doi:10.11959/j.issn.2096-6652.202230

Abstract

Abstract:

Combining batteries and super capacitors into a composite power system (CPS) with an effective energy management strategy can significantly improve the energy utilization, and increase the service life of the energy storage system.To minimize the energy loss of the system, an energy management strategy based on the twin delayed deep deterministic policy gradient (TD3) algorithm was designed.Compared with the deep deterministic policy gradient (DDPG) algorithm, TD3 algorithm solved the problem of overestimation of Q value and less energy loss.A MATLAB/Simulink simulation model based on the TD3 algorithm was built, and tested with the electric vehicle driving equation and the equivalent circuit model of CPS.The outcomes indicate that the proposed energy management strategy can effectively reduce the impact of high current on the battery, and compared with DDPG algorithm, the energy utilization efficiency is improved by 1.36%, the peak of output current of the battery is reduced by 14.68%, the temperature rise of the battery is reduced by 3.52%, the total energy consumption of the system is reduced by 2.17%.

Key words: electric vehicle, composite power system, energy management, deep reinforcement learning

CLC Number:

U469.72

Jiacheng LIU,Xiangwen ZHANG. TD3-based energy management strategy for hybrid energy storage system of electric vehicle[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 277-287.

Figures/Tables 16

References 24

[1]	崔淑梅, 宋贝贝, 王志远 . 电动汽车动态无线供电磁耦合机构研究综述[J]. 电工技术学报, 2022,37(3): 537-554.
	CUI S M , SONG B B , WANG Z Y . Overview of magnetic coupler for electric vehicles dynamic wireless charging[J]. Transactions of China Electrotechnical Society, 2022,37(3): 537-554.
[2]	张文爽, 李键, 余文涛 ,等. 基于新一代电源控制器的锂电池在轨管理分析[J]. 电源技术, 2022,46(1): 109-112.
	ZHANG W S , LI J , YU W T ,et al. Analysis and verification of in-orbit management of lithiumion battery based on the new generation of power control unit[J]. Chinese Journal of Power Sources, 2022,46(1): 109-112.
[3]	靳聪, 高申, 孙世光 ,等. 基于无损检测的生命周期锂离子动力电池安全性分析[J]. 储能科学与技术, 2019,8(2): 259-263.
	JIN C , GAO S , SUN S G ,et al. Analysis of safety performance of lithium-ion power battery during life cycle based on non-destructive testing[J]. Energy Storage Science and Technology, 2019,8(2): 259-263.
[4]	魏东涛, 黄之杰, 孔华 ,等. 蓄电池 SOC 的研究及预测方法[J]. 电源技术, 2016,40(6): 1321-1323.
	WEI D T , HUANG Z J , KONG H ,et al. Battery SOC research and prediction methods[J]. Chinese Journal of Power Sources, 2016,40(6): 1321-1323.
[5]	ZHAI C J , LUO F , LIU Y G . A novel predictive energy management strategy for electric vehicles based on velocity prediction[J]. IEEE Transactions on Vehicular Technology, 2020,69(11): 12559-12569.
[6]	ZHANG L J , YE X M , XIA X H ,et al. A real-time energy management and speed controller for an electric vehicle powered by a hybrid energy storage system[J]. IEEE Transactions on Industrial Informatics, 2020,16(10): 6272-6280.
[7]	周美兰, 冯继峰, 张宇 . 纯电动汽车复合储能系统及其能量控制策略[J]. 电机与控制学报, 2019,23(5): 51-59.
	ZHOU M L , FENG J F , ZHANG Y . Composite energy storage system and its energy control strategy for electric vehicles[J]. Electric Machines and Control, 2019,23(5): 51-59.
[8]	胡杰, 刘迪, 杜常清 ,等. 电动汽车复合能源系统能量管理策略研究[J]. 机械科学与技术, 2020,39(10): 1606-1614.
	HU J , LIU D , DU C Q ,et al. Study on energy management strategy of hybrid energy storage system for electric vehicles[J]. Mechanical Science and Technology for Aerospace Engineering, 2020,39(10): 1606-1614.
[9]	孙永健 . 某型纯电动客车复合电源系统优化设计与控制[D]. 长春:吉林大学, 2017.
	SUN Y J . The optimization design and control of hybrid energy storage system for A certain type electric bus[D]. Changchun:Jilin University, 2017.
[10]	CHEN Z Y , XIONG R , CAO J Y . Particle swarm optimization-based optimal power management of plug-in hybrid electric vehicles considering uncertain driving conditions[J]. Energy, 2016,96: 197-208.
[11]	ZHANG S , XIONG R , SUN F C . Model predictive control for power management in a plug-in hybrid electric vehicle with a hybrid energy storage system[J]. Applied Energy, 2017,185: 1654-1662.
[12]	VáZQUEZ-CANTELI J R , NAGY Z . Reinforcement learning for demand response:a review of algorithms and modeling techniques[J]. Applied Energy, 2019,235: 1072-1089.
[13]	XIONG R , CAO J Y , YU Q Q . Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle[J]. Applied Energy, 2018,211: 538-548.
[14]	WU J D , HE H W , PENG J K ,et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus[J]. Applied Energy, 2018,222: 799-811.
[15]	KONG H F , YAN J P , WANG H ,et al. Energy management strategy for electric vehicles based on deep Q-learning using Bayesian optimization[J]. Neural Computing and Applications, 2020,32(18): 14431-14445.
[16]	LI W H , CUI H , NEMETH T ,et al. Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles[J]. Journal of Energy Storage, 2021,36: 102355.
[17]	TAN H C , ZHANG H L , PENG J K ,et al. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space[J]. Energy Conversion and Management, 2019,195: 548-560.
[18]	WU Y K , TAN H C , PENG J K ,et al. Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus[J]. Applied Energy, 2019,247: 454-466.
[19]	SCHULMAN J , WOLSKI F , DHARIWAL P ,et al. Proximal policy optimization algorithms[J]. arXiv preprint,2017,arXiv:1707.06347.
[20]	SONG Z Y , HOU J , HOFMANN H ,et al. Sliding-mode and Lyapunov function-based control for battery/supercapacitor hybrid energy storage system used in electric vehicles[J]. Energy, 2017,122: 601-612.
[21]	QI X W , WU G Y , BORIBOONSOMSIN K ,et al. Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles[J]. Transportation Research Record:Journal of the Transportation Research Board, 2016,2572(1): 1-8.
[22]	LILLICRAP T P , HUNT J J , PRITZEL A ,et al. Continuous control with deep reinforcement learning[J]. arXiv preprint,2015,arXiv:1509.02971.
[23]	FUJIMOTO S , VAN HOOF H , MEGER D . Addressing function approximation error in actor-critic methods[J]. arXiv preprint,2018,arXiv:1802.09477.
[24]	丁镇涛, 邓涛, 李志飞 ,等. 基于安时积分和无迹卡尔曼滤波的锂离子电池 SOC 估算方法研究[J]. 中国机械工程, 2020,31(15): 1823-1830.
	DING Z T , DENG T , LI Z F ,et al. SOC estimation of lithium-ion battery based on ampere hour integral and unscented Kalman filter[J]. China Mechanical Engineering, 2020,31(15): 1823-1830.

Metrics

Recommended 0

No Suggested Reading articles found!

参数名称	描述	参数值
Actor网络层参数	各连接层神经元数量	(4,100,100,1)
Critic网络层参数	各连接层神经元数量	(5,100,100,1)
折扣因子	用于计算长期折扣奖励	0.95
Actor网络学习率	策略梯度更新步长	0.001
Critic网络学习率	Q函数网络梯度下降更新步长	0.001
优化算法类型	用于梯度下降	Adam
经验池尺寸	用于样本存储	10⁴
样本学习个数	一次批量梯度下降中的样本数量	64
软更新因子	目标网络参数更新速度	0.001
策略频率	用于延迟策略更新	2
策略噪声	添加在目标动作中的正态分布方差	0.2
裁剪幅度	指定裁剪的高斯噪声范围	(-0.5,0.5)
循环工况长度	所用UDDS长度	1 370
循环次数	训练次数	300

性能指标	单一电源	DDPG	PPO	TD3
蓄电池SOC剩余量	0.7019	0.7271	0.7320	0.7370
蓄电池峰值电流/A	107.61	74.23	68.77	63.33
蓄电池温升/℃	2.30	1.99	1.96	1.92
超级电容SOC剩余量	—	0.6606	0.6361	0.5418
系统总能耗/J	5.82×10⁶	5.08×10⁶	5.03×10⁶	4.97×10⁶

TD3-based energy management strategy for hybrid energy storage system of electric vehicle

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 24

Related Articles 10

Metrics

Recommended 0

[1]	Shuai MA, Qiming FU, Jianping CHEN, Fan FENG, You LU, Zhengwei LI, Shunian QIU. HVAC model-free optimal control method based on double-pools DQN [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(3): 426-444.
[2]	Yuxiang SUN, Yihui PENG, Bin LI, Jiawei ZHOU, Xinlei ZHANG, Xianzhong ZHOU. Overview of intelligent game:enlightenment of game AI to combat deduction [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 157-173.
[3]	Pu FENG, Wenjun WU, Jie LUO, Xin YU, Yongkai TIAN. Emergence measurement of robot swarm intelligence based on swarm entropy [J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(1): 65-74.
[4]	Zhiqiang HU. The framework model on internal mechanism of big data intelligent command and control [J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 101-109.
[5]	Zhaoyang LIU, Chaoxu MU, Changyin SUN. An overview on algorithms and applications of deep reinforcement learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 314-326.
[6]	Tao LI, Qinglai WEI. Intelligent heating temperature control system based on deep reinforcement learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 348-353.
[7]	Rizhong WANG, Huiping LI, Di CUI, Demin XU. Depth control of autonomous underwater vehicle using deep reinforcement learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 354-360.
[8]	Huiqiao FU, Kaiqiang TANG, Guizhou DENG, Xinpeng WANG, Chunlin CHEN. Motion planning for hexapod robot using deep reinforcement learning [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 361-371.
[9]	Yu SHEN,Jinpeng HAN,Lingxi LI,Fei-Yue WANG. AI in game intelligence—from multi-role game to parallel game [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(3): 205-213.
[10]	Hongye SU, Ze ZHOU, Zhitao LIU, Liyan Zhang. Research review and prospect of intelligent dynamic wireless charging system for electric vehicles [J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(1): 1-9.