基于TD3的电动汽车复合电源能量管理策略研究

doi:10.11959/j.issn.2096-6652.202230

摘要/Abstract

摘要：

将蓄电池与超级电容组成复合电源系统并结合有效的能量管理策略，能显著提高能量利用率，延长储能系统的使用寿命。为了实现复合电源系统能耗损失的最小化，设计了一种基于双延迟深度确定性策略梯度（TD3）算法的能量管理策略。与深度确定性策略梯度（DDPG）算法相比，该算法解决了Q值过高估计问题，能耗损失更小。利用电动汽车行驶方程式和复合电源系统等效电路模型，搭建了基于TD3算法的MATLAB/Simulink仿真模型，并进行测试。仿真结果显示，所提出的能量管理策略能降低大电流对蓄电池的冲击，与DDPG算法相比，能量利用率提高了1.36%，蓄电池峰值电流输出降低了14.68%，蓄电池温升降低了3.52%，系统总能耗降低了2.17%。

关键词: 电动汽车, 复合电源系统, 能量管理, 深度强化学习

Abstract:

Combining batteries and super capacitors into a composite power system (CPS) with an effective energy management strategy can significantly improve the energy utilization, and increase the service life of the energy storage system.To minimize the energy loss of the system, an energy management strategy based on the twin delayed deep deterministic policy gradient (TD3) algorithm was designed.Compared with the deep deterministic policy gradient (DDPG) algorithm, TD3 algorithm solved the problem of overestimation of Q value and less energy loss.A MATLAB/Simulink simulation model based on the TD3 algorithm was built, and tested with the electric vehicle driving equation and the equivalent circuit model of CPS.The outcomes indicate that the proposed energy management strategy can effectively reduce the impact of high current on the battery, and compared with DDPG algorithm, the energy utilization efficiency is improved by 1.36%, the peak of output current of the battery is reduced by 14.68%, the temperature rise of the battery is reduced by 3.52%, the total energy consumption of the system is reduced by 2.17%.

Key words: electric vehicle, composite power system, energy management, deep reinforcement learning

中图分类号:

U469.72

刘家成,张向文. 基于TD3的电动汽车复合电源能量管理策略研究[J]. 智能科学与技术学报, 2022, 4(2): 277-287.

Jiacheng LIU,Xiangwen ZHANG. TD3-based energy management strategy for hybrid energy storage system of electric vehicle[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 277-287.

图/表 16

图1

图2

图3

图4

图5

表1

图6

图7

图8

图9

图10

图11

图12

图13

图14

表2

参考文献 24

[1]	崔淑梅, 宋贝贝, 王志远 . 电动汽车动态无线供电磁耦合机构研究综述[J]. 电工技术学报, 2022,37(3): 537-554.
	CUI S M , SONG B B , WANG Z Y . Overview of magnetic coupler for electric vehicles dynamic wireless charging[J]. Transactions of China Electrotechnical Society, 2022,37(3): 537-554.
[2]	张文爽, 李键, 余文涛 ,等. 基于新一代电源控制器的锂电池在轨管理分析[J]. 电源技术, 2022,46(1): 109-112.
	ZHANG W S , LI J , YU W T ,et al. Analysis and verification of in-orbit management of lithiumion battery based on the new generation of power control unit[J]. Chinese Journal of Power Sources, 2022,46(1): 109-112.
[3]	靳聪, 高申, 孙世光 ,等. 基于无损检测的生命周期锂离子动力电池安全性分析[J]. 储能科学与技术, 2019,8(2): 259-263.
	JIN C , GAO S , SUN S G ,et al. Analysis of safety performance of lithium-ion power battery during life cycle based on non-destructive testing[J]. Energy Storage Science and Technology, 2019,8(2): 259-263.
[4]	魏东涛, 黄之杰, 孔华 ,等. 蓄电池 SOC 的研究及预测方法[J]. 电源技术, 2016,40(6): 1321-1323.
	WEI D T , HUANG Z J , KONG H ,et al. Battery SOC research and prediction methods[J]. Chinese Journal of Power Sources, 2016,40(6): 1321-1323.
[5]	ZHAI C J , LUO F , LIU Y G . A novel predictive energy management strategy for electric vehicles based on velocity prediction[J]. IEEE Transactions on Vehicular Technology, 2020,69(11): 12559-12569.
[6]	ZHANG L J , YE X M , XIA X H ,et al. A real-time energy management and speed controller for an electric vehicle powered by a hybrid energy storage system[J]. IEEE Transactions on Industrial Informatics, 2020,16(10): 6272-6280.
[7]	周美兰, 冯继峰, 张宇 . 纯电动汽车复合储能系统及其能量控制策略[J]. 电机与控制学报, 2019,23(5): 51-59.
	ZHOU M L , FENG J F , ZHANG Y . Composite energy storage system and its energy control strategy for electric vehicles[J]. Electric Machines and Control, 2019,23(5): 51-59.
[8]	胡杰, 刘迪, 杜常清 ,等. 电动汽车复合能源系统能量管理策略研究[J]. 机械科学与技术, 2020,39(10): 1606-1614.
	HU J , LIU D , DU C Q ,et al. Study on energy management strategy of hybrid energy storage system for electric vehicles[J]. Mechanical Science and Technology for Aerospace Engineering, 2020,39(10): 1606-1614.
[9]	孙永健 . 某型纯电动客车复合电源系统优化设计与控制[D]. 长春:吉林大学, 2017.
	SUN Y J . The optimization design and control of hybrid energy storage system for A certain type electric bus[D]. Changchun:Jilin University, 2017.
[10]	CHEN Z Y , XIONG R , CAO J Y . Particle swarm optimization-based optimal power management of plug-in hybrid electric vehicles considering uncertain driving conditions[J]. Energy, 2016,96: 197-208.
[11]	ZHANG S , XIONG R , SUN F C . Model predictive control for power management in a plug-in hybrid electric vehicle with a hybrid energy storage system[J]. Applied Energy, 2017,185: 1654-1662.
[12]	VáZQUEZ-CANTELI J R , NAGY Z . Reinforcement learning for demand response:a review of algorithms and modeling techniques[J]. Applied Energy, 2019,235: 1072-1089.
[13]	XIONG R , CAO J Y , YU Q Q . Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle[J]. Applied Energy, 2018,211: 538-548.
[14]	WU J D , HE H W , PENG J K ,et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus[J]. Applied Energy, 2018,222: 799-811.
[15]	KONG H F , YAN J P , WANG H ,et al. Energy management strategy for electric vehicles based on deep Q-learning using Bayesian optimization[J]. Neural Computing and Applications, 2020,32(18): 14431-14445.
[16]	LI W H , CUI H , NEMETH T ,et al. Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles[J]. Journal of Energy Storage, 2021,36: 102355.
[17]	TAN H C , ZHANG H L , PENG J K ,et al. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space[J]. Energy Conversion and Management, 2019,195: 548-560.
[18]	WU Y K , TAN H C , PENG J K ,et al. Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus[J]. Applied Energy, 2019,247: 454-466.
[19]	SCHULMAN J , WOLSKI F , DHARIWAL P ,et al. Proximal policy optimization algorithms[J]. arXiv preprint,2017,arXiv:1707.06347.
[20]	SONG Z Y , HOU J , HOFMANN H ,et al. Sliding-mode and Lyapunov function-based control for battery/supercapacitor hybrid energy storage system used in electric vehicles[J]. Energy, 2017,122: 601-612.
[21]	QI X W , WU G Y , BORIBOONSOMSIN K ,et al. Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles[J]. Transportation Research Record:Journal of the Transportation Research Board, 2016,2572(1): 1-8.
[22]	LILLICRAP T P , HUNT J J , PRITZEL A ,et al. Continuous control with deep reinforcement learning[J]. arXiv preprint,2015,arXiv:1509.02971.
[23]	FUJIMOTO S , VAN HOOF H , MEGER D . Addressing function approximation error in actor-critic methods[J]. arXiv preprint,2018,arXiv:1802.09477.
[24]	丁镇涛, 邓涛, 李志飞 ,等. 基于安时积分和无迹卡尔曼滤波的锂离子电池 SOC 估算方法研究[J]. 中国机械工程, 2020,31(15): 1823-1830.
	DING Z T , DENG T , LI Z F ,et al. SOC estimation of lithium-ion battery based on ampere hour integral and unscented Kalman filter[J]. China Mechanical Engineering, 2020,31(15): 1823-1830.

参数名称	描述	参数值
Actor网络层参数	各连接层神经元数量	(4,100,100,1)
Critic网络层参数	各连接层神经元数量	(5,100,100,1)
折扣因子	用于计算长期折扣奖励	0.95
Actor网络学习率	策略梯度更新步长	0.001
Critic网络学习率	Q函数网络梯度下降更新步长	0.001
优化算法类型	用于梯度下降	Adam
经验池尺寸	用于样本存储	10⁴
样本学习个数	一次批量梯度下降中的样本数量	64
软更新因子	目标网络参数更新速度	0.001
策略频率	用于延迟策略更新	2
策略噪声	添加在目标动作中的正态分布方差	0.2
裁剪幅度	指定裁剪的高斯噪声范围	(-0.5,0.5)
循环工况长度	所用UDDS长度	1 370
循环次数	训练次数	300

性能指标	单一电源	DDPG	PPO	TD3
蓄电池SOC剩余量	0.7019	0.7271	0.7320	0.7370
蓄电池峰值电流/A	107.61	74.23	68.77	63.33
蓄电池温升/℃	2.30	1.99	1.96	1.92
超级电容SOC剩余量	—	0.6606	0.6361	0.5418
系统总能耗/J	5.82×10⁶	5.08×10⁶	5.03×10⁶	4.97×10⁶